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I.  INTRODUCTION 


A.  BACKGROUND  DISCUSSION 

In  today’s  environment  of  aging  weapon  systems,  there  is  an  increased  need  for 
Operations  &  Support  (O&S)  funding.  However,  because  of  DoD  budgetary  constraints, 
there  has  been  a  trend  in  recent  years  to  utilize  discretionary  modernization  funds  in  an 
effort  to  fund  shortfalls  in  O&S  accounts.  As  a  result,  DoD’s  current  a;quisition 
approach  is  to  acquire  products  and  services  to  meet  military  needs  that  will  provide  the 
best  value  to  the  government  over  the  life  cycle  of  the  product  or  service.  Consequently, 
performance  parameters,  to  include  reliability  achievement,  must  be  considered  in 
relation  to  Total  Ownership  Costs  (TOC)  vice  simply  considering  the  initial  procurement 
costs  of  weapon  systems. 

Acquisition  decisions  made  early  in  the  life  cycle  of  weapon  systems  can  have  a 
tremendous  impact  on  the  availability  and  sustainment  of  Marine  Corps  equipment. 
Thus,  highly  reliable  systems  are  extremely  important  as  they  serve  as  force  effectiveness 
multipliers  that  significantly  contribute  towards  increased  system  availability,  a  reduced 
logistical  footprint,  and  a  net  reduction  in  total  ownership  costs,  which  equate  to 
increased  funds  for  modernization.  Therefore,  it  is  imperative  that  a  primary  goal  of 
systems  acquisitions  is  to  field  reliable  equipment  that  is  both  capable  and  supportable 
from  the  start. 

Both  the  inherent  reliability  designed  into  weapon  systems  and  the  estimates  of 
such  reliability  have  significant  impacts  on  weapon  system  readiness  and  cost  for  decades 
as  the  reliability  estimates  provide  the  basis  for  initial  life -cycle  supportability  decisions, 
including  integrated  logistical  support  packages.  Specifically,  such  estimates  contribute 
to  determining  the  initial  procurement  of  spare  parts  and  support  equipment,  concept  of 
logistical  support,  the  number  and  training  of  mechanics,  readiness  estimates,  operation 
and  support  costs,  and  Program  Objectives  Memorandum  (POM)  planning.  Therefore, 
the  effect  of  low  inherent  reliability,  as  well  as  the  effect  of  under  -  or  over-estimating  the 
reliability  of  weapon  systems,  will  cause  already  limited  dollars  to  be  allocated  unwisely 
as  unanticipated  life  cycle  costs  accumulate  and  cause  an  additional  need  for  O&S  dollars 
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in  later  years.  Consequently,  it  is  imperative  to  obtain,  verify,  and  utilize  accurate 
reliability  forecasts  early  in  the  life  cycle  process  and  to  attempt  to  tie  contractors  to 
readiness  and  LCC  thresholds  through  reliability  estimates. 

Fortunately,  there  are  many  early  opportunities  for  addressing  reliability  within 
weapon  systems  acquisitions.  Initially,  the  Requirements  Generation  Process  can  serve 
as  a  primary  tool  for  the  Marine  Corps  to  document  quantifiable  system  reliability 
requirements  in  the  Operational  Requirements  Document  (ORD)  in  the  form  of  Key 
Performance  Parameters  (KPP).  Additionally,  the  reliability  requirements  can  be  used  in 
source  selection  as  we  convert  specific  performance  specifications  into  contractual  terms, 
which  could  perhaps  include  an  inherent  reliability  goal.  From  here,  the  Systems 
Engineering  Process  allows  the  contractor  to  build  to  such  required  performance 
specifications.  Additionally,  once  contractors  submit  their  reliability  estimates,  program 
planning  and  organizational  management  can  emphasize  an  independent  and  rigorous 
reliability  testing  process  throughout  the  development  phase  in  order  to  demonstrate  th  e 
required  reliability  performance  levels  to  ensure  the  system  will  operate  in  the  field  as 
intended.  Lastly,  while  not  an  upfront  opportunity,  comparison  and  assessment  of 
achieved  field  reliability  to  contractor  reliability  estimates  could  be  conducted  throughout 
weapon  system  maturation  to  ensure  attainment  of  system  reliability  as  planned. 

However,  due  to  procedural,  management,  and  incentive  issues,  the  Marine  Corps 
is  faced  with  inhibitors  to  effective  reliability  management,  and  thus,  the  acquisit  ion 
community  has  not  been  able  to  fully  take  advantage  of  such  reliability  management 
opportunities.  Ultimately,  as  a  result,  the  warfighter  is  not  provided  with  a  reliable  and 
supportable  weapon  system  that  is  capable  of  being  sustained  within  its  life  cycle  cost 
threshold. 

B.  OBJECTIVES  AND  PURPOSE  OF  THE  RESEARCH 

The  purpose  of  this  research  is  to  evaluate  how  weapon  system  reliability 
performance  is  managed  throughout  the  acquisition  process  by  identifying  common 
inhibitors  and  enablers  of  effective  reliability  management,  why  they  occur,  lessons 
learned,  and  potential  methods  for  mitigating  the  inherent  risks.  The  results  of  the  thesis 
are  intended  to  directly  benefit  Program  Managers  while  providing  insight  into  the 

improved  sustainability  of  future  systems.  Understanding  that  reliability  estimates 
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provide  the  basis  for  initial  life  cycle  supportability  decisions,  the  acquisition  community 
must  utilize  effective  procedures,  as  well  as  develop  management  strategies  and 
techniques  to  address  reliability  risks.  The  research  ascertains  procedural  issues  that  the 
Marine  Corps  deals  with  concerning  reliability  requirements  in  the  acquisition  process  as 
well  as  common  management  and  incentive  issues  faced  by  program  management 
offices.  The  resulting  analysis  includes  conclusions  and  recommendations  applicable  to 
the  acquisition  community. 

C.  RESEARCH  QUESTIONS 

1.  Primary  Research  Question 

•  What  strategies  should  be  used  to  better  manage  weapon  system  reliability 
during  the  life  cycle  of  major  weapon  systems? 

2.  Subsidiary  Research  Questions 

•  How  does  reliability  affect  Life  Cycle  Cost  and  Operational  Availability? 

•  What  are  the  existing  policies,  regulations,  and  guidance  that  govern 
reliability  of  weapon  systems  available  to  the  Combat  Developer,  Program 
Management  Office,  and  Contractor?  Do  they  provide  adequate 
guidance? 

•  How  does  the  Marine  Corps  address  reliability  performance  of  weapon 
systems  during  the  Requirements  Generation  Process? 

•  How  can  the  Marine  Corps  create  and  adhere  to  a  contractual  oblig  ation  in 
the  form  of  quantitative  system  reliability  requirements  that  forces 
contractors  to  consider  reliability  equally  with  other  system  parameters 
such  cost,  schedule,  and  performance? 

•  How  is  system  reliability  addressed  during  developmental  and  oper  ational 
testing,  and  is  the  Marine  Corps  adequately  testing  to  determine  and 
demonstrate  the  required  reliability  performance  levels? 

•  Is  there  a  significant  difference  between  contractors’  reliability  estimates 
and  achieved  reliability  of  fielded  systems  as  obtained  from  Marine  Corps 
logistics  systems,  and  if  so,  is  the  Marine  Corps  adequately  assessing  the 
data  during  the  maturation  of  weapon  systems  in  order  to  alleviate  future 
contractor  reliability  inaccuracies? 

•  Is  the  Marine  Corps  maintenance  rate,  in  the  form  of  MTBM,  a  feasible 
surrogate  for  comparison  with  traditional  failure  rate,  in  the  form  of 
MTBF,  as  obtained  from  contractors? 

D.  SCOPE,  LIMITATIONS,  AND  ASSUMPTIONS 

The  research  for  this  thesis  was  completed  in  collaboration  with  a  similar 
concurrent  study  conducted  by  Studies  and  Analysis  Division,  Marine  Corps  Combat 
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Development  Command  (MCCDC)  under  the  sponsorship  of  Marine  Corps  Systems 
Command  (MARCORSYSCOM),  entitled  “Sustainment  Consequences  of  Acquisition 
Decisions.”  The  thesis  assesses  pre-fielding  programmatic  and  technical  decisions  that 
influence  reliability  of  fielded  systems  (MARCORSYSCOM  study  plan  “Sustainment 
Consequences  .  .  Specifically,  the  scope  of  this  research  includes  an  evaluation  of 
reliability  management  within  the  Marine  Corps  acquisitions  process  from  numerous 
perspectives  to  include:  1)  a  review  of  the  relationship  between  reliability,  operational 
availability,  logistics  support,  and  life-cycle  costs,  2)  a  review  and  assessment  of  current 
DoD  and  Marine  Corps  policy,  guidance,  and  regulations  regarding  reliability,  3)  an 
examination  of  reliability  requirements  documentation  and  its  relevance  in  source 
selection,  4)  an  assessment  of  transforming  ORD  reliability  performance  specifications 
into  contractual  obligations,  5)  an  evaluation  of  the  extent  to  which  reliability 
requirements  are  being  demonstrated  during  testing,  6)  a  comparison  and  assessment  of 
reliability  requirements  and  contractor  reliability  estimates  to  actual  achieved  reliability 
of  fielded  systems,  and  7)  an  analysis  of  the  Marine  Corps’  adequacy  of  comparing  and 
assessing  the  aforementioned  data  during  the  maturation  of  weapon  systems.  The 
research  will  aide  in  assessing  the  accuracy  and  completeness  of  reliability  estimates  for 
fielded  systems  while  identifying  techniques  to  improve  the  accuracy  of  reliability 
estimates  during  systems  development.  Furthermore,  a  comparison  of  documented 
reliability  requirements  and  pre-fielding  estimates  to  achieved  reliability  will  provide 
beneficial  insight  into  achieving  future  readiness  (MARCORSYSCOM  Draft  SOW 
“Sustainment  Consequences  .  .  .”). 

The  data  collected  is  limited  to  mature  Critical/Pacing  items  included  in  the 
Quarterly  Readiness  Reports  to  Congress  (M1A1  tank,  AAV  family  of  vehicles,  LAV 
family  of  vehicles,  5ton  truck  family  of  vehicles,  HMMWV  family  of  vehicles,  LVS 
family  of  vehicles,  and  the  M198  Howitzer).  The  analysis  is  limited  to  an  assessment  of 
reliability  management  issues,  while  not  specifically  addressing  technology  driven 
reliability  problems.  While  the  research  is  limited  to  selected  principle  end  items,  it  is 
assumed  that  the  challenges,  issues,  and  potential  solutions  can  be  applied  to  other  end 
items  in  the  Marine  Corps  acquisition  process. 
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E  METHODOLOGY 

In  an  effort  to  determine  the  current  environment  for  reliability  management 
within  Marine  Corps  acquisitions,  the  researcher  administered  an  electronic  survey 
(Appendix  B)  to  various  personnel  within  the  Program  Offices  of  specific  critical/  pacing 
end  items.  The  questions  pos  ed  were  intended  to  emphasize  the  perspective  of  program 
management  leadership  on  the  varied  tasks  involved  with  reliability  management 
(Masiello,  p.  4).  Specifically,  the  survey  was  intended  to  conduct  an  examination  of 
current  policy  and  regulations,  reliability  requirements  documentation,  contractual 
obligations,  developmental  and  operational  test  data,  and  readiness/maintenance  data. 
The  survey  utilized  was  a  modification  of  a  previously  designed  reliability  performance 
survey  intended  to  gather  data  within  a  specific  Army  Program  Executive  Office  in 
pursuit  of  similar  research  objectives  (Ryan,  pp.  91-97). 

The  methodology  used  in  this  thesis  research  consisted  of  the  following  steps: 

•  Through  a  review  of  existing  publications,  examine  and  document  the 
relationship  between  reliability,  logistics,  life-cycle  support  costs,  and 
readiness 

•  Review  and  examine  the  adequacy  of  current  DoD  and  Marine  Corps 
policy,  guidance,  and  regulations  that  govern  reliability 

•  Conduct  a  review  of  the  acquisition  process,  from  determining  needs 
requirement  through  sustainment  operations  and  support 

•  Through  the  combination  of  data  collection  from  the  Fleet  and  reliability 
survey  responses  from  the  acquisition  community: 

•  Determine  the  extent  to  which  the  Marine  Corps  organizations 
involved  throughout  the  acquisition  process  consider  reliability 

•  Determine  how  the  Marine  Corps  addresses  reliability  performance 
in  the  requirements  generation  phase 

•  Review  the  current  process  and  methods  of  transforming  ORD 
requirements  into  quantifiable  contractual  obligations 

•  Determine  the  extent  to  which  reliability  requirements  are 
demonstrated  during  testing 

•  Determine  if  contractor  reliability  estimates  are  retained,  and 
determine  the  achieved  reliability  data  of  mature  fielded  systems 

•  Compare  and  assess  the  predetermined  reliability  requirements  and 
contractor  estimates  to  achieved  reliability  of  mature  systems. 
Determine  and  evaluate  the  Marine  Corps’  adequacy  at  conducting 
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the  same  comparison  throughout  the  maturation  of  weapon 
systems. 

•  Assess  the  collected  data  to  identify  policy,  managerial,  and 
procedural  issues  involved  with  current  reliability  management  in 
the  acquisition  process 

•  Recommend  policy  and  procedural  changes  to  reliability 
management  throughout  the  acquisition  process  and  provide 
insight  into  the  improved  sustainability  of  future  systems  through 
the  obtainment  of  accurate  reliability  estimates  from  contractors 

F.  ORGANIZATION  OF  THE  RESEARCH 

This  thesis  contains  six  chapters. 

Chapter  I  introduces  the  subject  of  reliability  as  a  basis  for  the  study  while 
providing  the  objectives,  scope,  methodology,  organization,  and  benefits  of  the  research. 

Chapter  II  provides  a  background  and  overview  of  reliability  while  defining 
reliability  and  related  concepts.  The  relationship  between  reliability,  logistics,  life-cycle 
support  costs,  and  operational  availability  will  be  addressed.  Additionally,  this  chapter 
discusses  the  tools  and  techniques  available  for  reliability  analysis. 

Chapter  III  is  a  brief  overview  of  the  acquisition  process  from  the  Requirements 
Generation  Process  through  Sustainment  Operations  and  Support.  Additionally,  this 
chapter  discusses  the  participants  and  organizations  involved  in  the  process.  Also,  the 
current  DOD  and  Marine  Corps  policies,  regulations,  and  guidanc  e  that  establish  the 
basis  within  which  the  acquisition  community  should  operate  to  manage  reliability  within 
a  program  will  be  discussed. 

Chapter  IV  provides  the  program  demographics  and  background  of  the  systems 
that  are  a  part  of  this  study  and  presents  the  aggregate  results  of  the  data  collection  from 
the  reliability  survey.  This  data  indicates  how  the  respective  programs  have  implemented 
reliability  management  processes  and  highlights  significant  examples  and  experiences. 

Chapter  V  analyzes  and  compiles  the  key  issues  and  challenges  associated  with 
reliability  to  include  issues  with  existing  policy  and  guidance  on  reliability,  reliability 
requirements  determination  and  documentation,  contracting  for  reliability,  developmental 
and  operational  testing,  and  comparison  and  assessment  of  reliability  requirements  and 
estimates  to  achieved  reliability. 
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The  final  chapter  makes  conclusions  and  recommendations,  provides  answers  to 
the  primary  and  secondary  research  questions,  and  recommends  areas  for  further 
research. 

G.  BENEFITS  OF  THE  RESEARCH 

According  to  Marine  Corps  Systems  Command,  there  are  currently  no  known 
studies  within  the  Marine  Corps  comparing  the  relationship  of  reliability,  availability,  and 
maintainability  (RAM)  to  Operational  Availability  and  determining  its  impact  on  Future 
Readiness  thresholds  (MARCORSYSCOM  Draft  SOW  “Sustainment  Consequences  .  . 

p.  3).  Thus,  the  primary  benefit  of  this  study  is  the  identification  of  policy  and 
program  management  issues  with  respect  to  weapon  system  reliability  and  providing 
recommendations  for  areas  of  potential  improvement.  The  research  is  intended  to 
directly  benefit  the  acquisition  community  by  identifying  common  potential  inhibitors, 
identifying  their  underlying  root  causes,  providing  lessons  learned,  and  suggesting 
methods  for  managing  and  reducing  inherent  risks  associated  with  achieving  reliability 
performance  requirements.  Additionally,  attaining  accurate  contractor  reliability 
estimates,  used  as  a  basis  for  initial  life  cycle  supportability  issues,  will  benefit  the 
Marine  Corps  by  optimizing  the  use  of  constrained  resources  and  improving  the 
operational  force  materiel  readiness  posture  (MARCORSYSCOM  Draft  SOW 
“Sustainment  Consequences  .  .  .”)■ 
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II.  RELIABILITY  OYER  VIEW  AND  BACKGROUND 


When  in  a  fight  to  the  death,  one  wants  to  employ  all  one’s  weapons  to  the 
utmost.  I  must  say  that  to  die  with  one’s  sword  still  sheathed  is  most 
regrettable.  -  Miyamoto  Musashi,  Book  of  Five  Rings 

A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  provide  a  fundamental  understanding  of 
reliability  and  its  importance  within  weapon  systems  acquisitions.  This  will  be 
accomplished  by  addressing  the  relationship  between  reliability,  logistics,  life-cycle 
support  costs,  and  operational  availability.  However,  an  overview  of  reliability  and 
related  concepts  will  first  be  required  to  provide  a  common  frame  of  reference  and 
establish  a  general  basis  of  understanding  for  subsequent  discussions.  Accordingly,  this 
chapter  also  discusses  the  alignment  of  process  ownership  between  the  Program 
Management/Weapon  System  Management  (PM/WSM)  and  Supply  Chain  Management 
(SCM)  organizational  elements  while  detailing  the  changes  recently  implemented  within 
the  Marine  Corps  to  best  accommodate  life  cycle  management  of  its  equipment.  Lastly, 
tools  available  for  reliability  analysis  will  be  briefly  introduced. 

B.  RELIABILITY  DEFINED :  RELATED  DEFINITIONS,  CONCEPTS,  AND 
MEASURES 

In  order  to  address  the  role  of  reliability  in  the  logistics  community,  it  is 
imperative  to  understand  the  terms  and  definitions  most  widely  associated  with  defining 
and  discussing  reliability.  The  intent  of  this  section  is  to  provide  basic  quantitative  and 
qualitative  knowledge  of  reliability -related  definitions  and  concepts  required  to  plan  for, 
design,  produce,  and  implement  an  effective  and  efficient  logistic  support  capability.  Of 
particular  emphasis  within  weapon  systems  acquisitions  are  the  qualitative  measures  of 
reliability  and  logistics,  which  must  be  addressed  in  order  to  ensure  logistics 
requirements  are  adequately  specified,  evaluated,  and  modified  for  improvement.  In 
addition  to  reliability  itself,  other  measurements  are  utilized  to  characterize  the  reliability 
of  a  system  and  its  components. 
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1.  Reliability 

The  probability  that  a  system  or  product  will  perform  in  a  satisfactory 

manner  for  a  given  period  of  time  when  used  under  specified  operating 

conditions  (Blanchard,  p.  25). 

•  When  considering  component  reliability,  the  term  “system”  can  be 
extended  to  include  components  or  subsystems  that  can  be  considered  as 
an  entity 

•  The  term  “satisfactory”  indicates  that  specific  criteria  must  be  established 
to  determine  what  satisfactory  operation/service  is 

•  For  a  hardware  item  to  be  reliable  it  must  do  more  than  meet  an  initial 
factory  performance  requirement  -  it  must  operate  for  a  given  period  of 
time  in  the  actual  application  for  whic  h  it  is  intended.  “Time”  represents  a 
measure  against  which  the  degree  of  system  performance  can  be  related. 

Inherent  reliability  is  the  potential  reliability  of  a  system  (inherent  as 

designed),  assuming  an  ideal  operating  and  support  environment. 

As  evident  from  the  preceding  clarifications,  the  concept  of  reliability  is  often 
utilized  without  precise  definition,  while  the  terminology  is  non-standard  throughout  the 
logistics  community  and  tends  to  depend  on  the  Service  and/or  system.  In  the  broadest 
sense,  reliability  is  associated  with  dependability,  with  successful  operations,  and  with 
the  absence  of  breakdowns  or  failures  (Lewis,  p.  1).  However,  while  creating  DoD 
requirements  documentation  and  contract  specifications,  it  is  very  important  that  all  ma  in 
concepts  are  addressed  in  an  unambiguous  way  so  that  all  parties  involved  understand  the 
terms.  Furthermore,  to  adequately  conduct  engineering  analyses,  reliability  must  be 
defined  quantitatively  as  a  probability.  Thus,  one  must  consider  the  time  parameter  in 
order  to  assess  the  probability  of  completing  a  given  function  as  scheduled.  The 
reliability  function,  R(t),  may  be  expressed  as: 

R(t)  =  Pr (T  >  t)  =  J” /(f) dt  =  1  -Fit)  (2. 1) 

Let  T  be  a  random  variable  that  represents  the  time  until  the  next  failure,  f(t)  be 
the  probability  density  function,  and  F( t)  be  the  cumulative  density  function  of  T. 

Then  the  reliability  function,  R( t),  is  defined  as  the  probability  that  the  failure  will 
not  occur  until  time  t. 
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Assuming  that  the  time  to  failure  is  described  by  an  exponential  density  function, 
the  reliability  function,  R(t),  is: 

R(t)  =  Pr(T  >t)  =  j'f(t)dl  =  \  -F(t)  =  A<?  L'dx  =  e  lJ  (2.2) 

where  t  is  the  time  period  of  interest,  and  e  is  the  natural  logarithm  base  (2.7183),  and  K 
is  the  instantaneous  failure  rate  (Blanchard,  p.  37).  It  is  important  to  note  that  the 
reliability  function  as  depicted  above  is  in  terms  of  an  exponential  distribution.  This 
means  that  the  unit’s  failure  rate  is  constant  over  the  period  t,  the  reliability  for  a  new 
mission  is  independent  of  the  age  of  the  unit  and  is  a  function  of  its  failure  rate  and  the 
duration  of  the  new  mission  only.  This  is  commonly  used  in  many  applications  under  the 
presumption  that  all  like  components  are  being  utilized  in  the  exact  same  manner  with  the 
same  stresses  imposed  upon  them.  In  reality,  the  failure  characteristics  of  different 
components  vary  considerably  depending  upon  their  usage.  Other  applicable  dens  ity 
functions  include  the  normal,  binomial,  exponential,  Poisson,  gamma,  and  Weibull 
distributions  (Kececioglu,  p.  202).  However,  explanation  of  such  distributions  are 
beyond  the  scope  of  this  thesis. 

2.  Failure  Rate 

The  number  of  item  failures  of  per  measure  of  unit  life,  where  failure  is 
defined  as  the  termination  of  an  item’s  ability  to  perform  a  required 
function  (Hoyland  and  Rausand,  p.  10). 

The  failure  rate  is  expressed  as: 

.  numberof  failures  1 

A,  = - = -  (2.3) 

totaloperatinghours  MTBF 

When  determining  overall  failure  rate,  it  is  important  to  address  all  system  factors 
that  cause  the  system  to  be  inoperative  at  a  time  when  satisfactory  system  operation  is 
required.  A  combined  failure  rate  is  presented  in  Table  2.1. 
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Consideration 

Assumed  Factor 
(instances/hour) 

(a)  Inherent  reliability  failure  rate 

.000392 

(b)  Manufacturing  defects 

.000002 

(c)  Wear-out  rate 

.000000 

(d)  Dependent  failure  rate 

.000072 

(e)  Operator-induced  failure  rate 

.000003 

(f)  Maintenance-induced  failure  rate 

.000012 

(g)  Equipment  damage  rate 

.000005 

Total  combined  factor 

.000486 

Table  2.1.  Combined  Failure  Rates.  (From:  Blanchard,  p.  40) 


3.  Mean  Time  Between  Failure  (MTBF) 

For  a  particular  interval,  the  total  functional  life  of  a  population  of  an  item 
divided  by  the  total  number  of  failures  with  the  population  (DSMC, 
“Acquisition  Logistics  Guide”,  p.  10-2). 


MTBF  serves  as  the  basic  technical  measure  of  reliability,  and  thus,  the  measure 
becomes  a  key  element  in  support  planning.  In  simplified  terms,  MTBF  is  the  average 
time  between  required  corrective  (unscheduled)  maintenance  actions.  MTBF  should  not 
be  used  interchangeably  with  failure  rate,  and  in  fact,  MTBF  is  the  inverse  of  the  failure 
rate: 


MTBF  =  — 

X 


(2.4) 


It  is  important  to  distinguish  why  MTBF  needs  to  be  calculated  for  equipment. 
The  calculation  of  this  time  is  necessary  in  order  to  determine  whether  the  mean  time 
between  failures  is  increasing,  decreasing,  or  remaining  constant  with  age.  As  equipment 
ages,  its  MTBF  decreases  until  the  cost  of  keeping  that  item  operational  is  more  than  the 
cost  of  buying  a  new  item.  Estimates  of  when  maintenance  costs  will  exceed  acquisition 


12 


costs  are  questionable  without  mean  time  between  failure  calculation  (Enholm,  p.  1).  In 
other  words,  MTBF  data  analysis  can  help  to  determine  if  equipment  is  in  the  “wear-out” 
phase  of  its  life  cycle  and  at  the  end  of  its  economic  useful  life. 

4.  Mean  Time  Between  Maintenance  (MTBM) 

MTBM  includes  both  preventive  (scheduled)  and  corrective  (unscheduled) 
maintenance  requirements.  It  includes  consideration  of  reliability  MTBF  and  MTBR. 
MTBM  may  also  be  considered  as  a  reliability  parameter  and  can  be  expressed  as: 

MTBM= - - - - - t - =~ — —  (2.5) 

_ 1 _ |  1  X +  fpt 

MTBM  . , , .  MTBM  .... 

unscheduled  scheduled 

wher efpt  (=1/MTBMS)  is  the  frequency  of  the  preventive  maintenance  actions  per  system 
operating  hour,  or  the  preventive  maintenance  rate.  Also,  MTBM  unscheduied  (same  as 
MTBF)  is  the  mean  interval  of  unscheduled  maintenance  and  MTBM  scheduled  is  the  mean 
interval  of  scheduled  maintenance  (NPS  Fogistics  Engineering  principle). 

It  should  be  obvious  that  MTBM  is  not  the  same  measurement  as  MTBF  due  to 
the  inclusion  of  preventive  maintenance  actions.  However,  the  Marine  Corps  is  often 
forced  to  substitute  MTBF  with  MTBM  due  to  lack  of  operational  usage  data  needed  to 
calculate  MTBF.  The  feasibility  of  this  substitution  will  be  discussed  in  more  thorough 
detail  later  in  the  thesis. 

5.  Availability 

The  probability  that  an  item  (system)  is  in  an  operable  and  committable 
state  at  the  start  of  a  mission  when  the  mission  is  called  for  at  a  random 
point  in  time.  “Is  the  equipment  available  in  a  working  condition  when  it 
is  needed?”  (DSMC,  “Acquisition  Fogistics  Guide”,  p.  10-3) 

Availability  is  frequently  used  as  a  measure  of  system  readiness,  and  thus,  the 
user  is  often  most  concerned  about  this  parameter.  There  are  numerous  expressions  of 
availability,  all  of  which  are  based  on  the  standard  mathematical  relationship  between 
“up  time”,  “down  time”,  and  “total  time.”  In  other  words,  over  long  operating  periods, 
availability  can  essentially  be  expressed  as  a  relationship  between  uptime  (reliability)  and 
downtime  (DSMC,  “Designing  Quality  into  .  .  .”,  p.  B-l). 
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a.  Inherent  Availability  (Aj 

Inherent  availability  takes  into  account  only  items  of  systems  design. 
Additionally,  it  assumes  an  ideal  support  environment  and  includes  only  active  corrective 
maintenance  time  in  calculation  of  downtime  while  excluding  preventive  maintenance 
time  and  servicing  times  as  well  as  supply,  administrative  and  personnel  delays.  Inherent 
availability  is  expressed  in  terms  of  its  designed  mean  time  between  failures  (MTBF)  and 
its  designed  mean  time  to  repair  (or  active  repair  time)  (MTTR)  given  that  it  has  failed: 

MTBF  MTBF 


A=- 


(MTBF  +  MTTR)  ( MTBF  +MJ 


(2.6) 


where  Mct  =  mean  corrective  maintenance  time. 

I).  Achieved  Availability  (Aa) 

Achieved  availability  is  calculated  when  preventive  maintenance  is 
included  in  the  relationship.  However,  an  ideal  (no  delay)  support  system  is  still 
assumed,  which  excludes  Logistics  Delay  Time  (LDT)  and  Administrative  Delay  Time 
(ADT): 

MTBM 


A=- 


MTBM+M 


(2.7) 


where  M  =  mean  active  maintenance  time  (both  preventive  and  corrective  maintenance 
activities)  and  MTBM  is  the  mean  time  between  maintenance,  both  corrective  and 
preventive. 

c.  Operational  Availability  (A0) 

Operational  Availability  is  a  function  of  the  reliability  and  maintainability 
of  the  equipment  and  is  a  commonly  used  measure  of  weapon  system  readiness.  It  is  the 
most  desirable  form  of  availability  to  be  used  in  helping  assess  a  system’s  potential  under 
fielded  conditions  whereas  achieved  availability  and  inherent  availability  are  primarily 
the  concern  of  the  developing  organization  in  its  interface  with  the  contractor  (DSMC, 
“Acquisition  Logistics  Guide”,  p.  104).  Specifically,  operational  availability  is  the 
probability  that  a  system,  when  used  under  stated  conditions  in  an  actual  operational 
environment,  will  operate  satisfactorily  when  called  upon  at  any  random  time. 
Additionally,  operational  availability  includes  all  of  the  sources  of  non-operable  time, 
active  and  inactive  to  include  supply  and  administrative  delay  times,  corrective  and 
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preventive  maintenance,  and  personnel/maintenance  technician  delays.  The  value 
provides  both  the  percentage  of  time  that  a  system  is  in  a  mission  capable  status  in  the 
long-run  and  the  percentage  of  weapon  systems  in  mission  capable  status: 

^  MTBM  Numberof  MissionCapableSystems  ^ 

MTBM+MDT  Totalnumberof  systems 

where  MDT  =  maintenance  downtime,  or  the  total  elapsed  time  required  to  repair  and 
restore  a  system  to  full  operating  status.  Maintenance  downtime  (MDT)  includes  mean 
active  maintenance  (M),  logistics  delay  time  (LDT),  and  administrative  delay  time 
(ADT). 

Despite  which  expression  of  availability  used,  it  is  obvious  that  reliability 
is  a  major  driver  in  the  numerator  of  these  relationships. 

6.  Reliability  Component  Relationships 

Overall  system  reliability  is  a  function  of  the  reliability  of  subsystems  and 
components.  With  today’s  technology,  systems  performance  may  often  be  increased  at 
the  expense  of  increased  complexity;  the  complexity  usually  being  measured  by  the 
number  of  required  components  and  parts.  However,  unless  compensating  measures  are 
taken  to  improve  the  reliability  of  the  components,  system  reliability  will  decrease.  This 
is  because  if  nothing  else  is  changed,  reliability  decreases  with  each  added  component. 
In  such  cases  of  increased  system  complexity,  reliability  can  only  be  maintained  if 
component  reliability  is  increased  or  if  component  redundancy  is  built  into  the  system. 
However,  each  of  these  solutions,  in  turn,  must  be  measured  against  incurred  costs 
(Lewis,  p.  3). 

The  decrease  in  reliability  due  to  increased  system  complexity  may  be  expressed 
in  terms  of  the  product  rule.  The  reliability  of  the  system  is  the  product  of  reliabilities  of 
the  individual  subcomponents.  In  other  words,  if  the  component  failures  are  mutually 
independent  in  a  series  form,  the  reliability  of  the  system  with  N  nonredundant 
components  is: 

R  =  RlRY..Rn...RN  (2.9) 

As  depicted,  in  a  series  network,  all  components  must  operate  in  a  satisfactory 
manner  if  the  system  is  to  function  properly.  Connecting  subsystems  in  a  series  tends  to 
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decrease  reliability,  since  the  reliability  of  the  entire  system  is  equal  to  the  product  of  the 
individual  reliabilities  of  that  system. 

However,  from  a  reliability  perspective,  system  components  can  be  integrated  in 
parallel  form,  enabling  system  developers  to  increase  system  reliability  through  increased 
redundancy  in  the  system.  In  a  parallel  network,  a  number  of  the  same  components  are  in 
parallel,  and  thus,  all  components  must  fail  in  order  to  cause  total  system  failure.  For  a 
system  with  n  identical  components,  the  reliability  expression  for  the  system  is: 

R=l-(l-R)n  (2.10) 

Parallel  redundant  networks  are  used  primarily  to  improve  system  reliability 
(Blanchard,  p.  45).  Additionally,  various  levels  of  reliability  can  be  achieved  through  the 
application  of  combining  series  and  parallel  networks.  In  fact,  a  combination  of  both 
types  of  systems  is  commonplace  and  almost  unavoidable.  Once  systems  engineers 
determine  the  reliability  of  individual  components,  overall  system  reliability  can  be 
empirically  calculated.  Ultimately,  the  true  source  of  system  reliability  rests  with  the 
performance  of  individual  components  and  subsystems  (Chaudhary,  p.  26). 

7.  Reliability  Bathtub  Curve 

The  reliability  of  a  system  and  its  components  will  fluctuate  throughout  their 
development,  production  life  cycle,  and  operational  usage.  Additionally,  product  updates, 
system  changes  or  modifications,  and  maintenance  actions  further  affect  the  reliability  of 
systems  and  their  components.  However,  assuming  a  negative  exponential  distribution, 
the  failure  rate  is  relatively  constant  during  the  mature  stages  of  a  system  life  cycle  as 
shown  in  Figure  2.1.  It  is  during  this  relatively  stable  portion  of  the  curve  that  the 
exponential  failure  law  applies.  However,  when  systems  are  initially  operational,  there 
are  usually  a  higher  number  of  failures  mostly  attributable  to  poor  manufacturing 
techniques,  poor  quality  control,  poor  workmanship,  insufficient  burn-in  or  break-in, 
improper  installation,  insufficient  debugging,  human  error,  and  other  causes.  As  a  result, 
the  initial  failure  rate  is  higher  than  anticipated  before  leveling  off  to  the  constant  failure - 
rate  region.  Likewise,  when  a  system  reaches  a  certain  age,  it  enters  its  wear  -out  life 
period  where  the  failure  rate  once  again  increases  (Kececioglu ,  p.  74).  It  should  be  noted 
that  the  curve  would  vary  depending  upon  the  type  of  system  and  its  operational  usage. 
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Figure  2.1.  Reliability  Bathtub  Curve.  (From:  Kececioglu,  p.  74) 


Effective  reliability  programs  require  the  assessment  of  reliability  at  key  decision 
points  along  the  growth  curve.  Data  availability  for  making  projections  obviously 
increases  as  the  program  and  its  tests  progress.  For  example,  during  the  early  life  period, 
known  as  the  infant  mortality  period,  reliability  estimates  must  be  made  on  information 
obtained  from  stress  calculations,  proven  component  data  from  similar  equipment, 
accelerated  testing,  and  potential  problem  analysis,  all  of  which  are  reliability  analysis 
tools  to  be  discussed  later  in  this  Chapter. 

Ultimately,  the  actual  reliability  level  of  a  system  and  its  components,  as  well  as 
the  confidence  in  the  estimated  level,  increases  with  the  test  program  and  its 
corresponding  corrective  actions.  Attempts  must  be  made  to  obtain  the  required  times-to- 
failure  and  success -and-failure  data  in  an  effort  to  prepare  a  reliability  bathtub  curve, 
plotting  the  failure  rate  of  a  system  versus  its  age.  Such  a  curve  enables  the  estimation  of 
(a)  the  optimum  break-in  testing  period  and  burn-in  time,  (b)  the  optimum  warranty  time 
and  its  cost,  (c)  the  optimum  preventive  replacement  time,  and  (d)  the  spares 
requirements  (Kececioglu,  “Reliability  Engineering  Handbook”,  p.  37). 
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C.  RELATIONSHIP  BETWEEN  RELIABILITY,  LOGISTICS,  LIFE  CYCLE 
SUPPORT  COSTS  AND  READINESS 

Early  materiel  life  cycle  decisions  during  the  acquisition  process  have  a 
significant  impact  on  future  operational  availability  and  life  cycle  cost  of  weapon 
systems.  This  is  largely  due  to  the  fact  that  reliability  characteristics  that  are  inherent 
within  the  system  design  actually  dictate  the  requirements  for  the  subsequent 
maintenance  and  support  of  that  system  throughout  its  life  cycle  (Blanchard,  p.  252).  In 
addition  to  actual  inherent  reliability  associated  with  system  design,  under-  or  over- 
estimations  of  the  reliability  of  weapon  systems  in  development  dramatically  and  often 
adversely  affect  life  cycle  cost  and  operational  availability  as  the  reliability  estimate 
provides  the  basis  for  initial  life  cycle  supportability  decisions. 

Weapon  systems  must  be  designed  to  be  supportable  for  the  warfighter,  capable 
of  being  maintained  effectively  and  efficiently  throughout  their  planned  life  cycles, 
ultimately  enabling  the  warfighter  to  focus  his  efforts  on  his  primary  task  of  winning 
battles  and  providing  him  with  equipment  capable  of  doing  so.  Therefore,  the  DOD  must 
remain  focused  on  the  goal  of  providing  systems  that  maximize  their  operational 
availability  (A0)  within  the  allocated  life-cycle  cost  (LCC)  of  the  program.  When 
considering  readiness  and  supportability  objectives  within  budgetary  constraints,  system 
reliability  emerges  as  the  prominent  life  cycle  cost  and  readiness  driver  for  defense 
weapons  systems.  Thus,  it  is  critical  to  consider  the  role  of  reliability  in  planning  for 
integrated  logistical  support  in  the  early  stages  of  planning  and  design  as  well  as 
throughout  the  entire  acquisition  process.  However,  before  attempting  to  specify 
quantitative  reliability  requirements  and  considering  managerial  or  procedural  methods  to 
improve  reliability,  one  must  be  able  to  clearly  establish  the  link  between  reliability,  life 
cycle  cost,  and  readiness. 

1.  Impact  of  Reliability  on  Operational  Availability 

The  ability  to  successfully  complete  a  mission  is  directly  dependent  on  the 
weapon  performing  that  mission  without  experiencing  a  mission  critical  failure.  In  other 
words,  weapon  system  reliability  directly  affects  the  ability  of  the  Marine  Corps  to 
perform  its  mission.  With  this  in  mind,  it  becomes  clear  that  “reliability  isn’t  everythin  g, 
it  is  the  only  thing”  (Eaton  Email,  25  April  2001). 
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The  following  formula  indicates  that  there  is  a  definite  direct  relationship  between 
reliability,  maintainability,  and  readiness  (A0): 


uptime  _  MTBM 
uptime  +  dowtime  MTBM  +MDT 


UPTIME 

_ OT+ST _ 

OT  +  ST  +  ALDT  +CMT  +PMT 

UPTIME  DOWNTIME 


(2.11) 


where, 

OT  =  Operating  Time 
ST  =  Standby  Time 

ALDT  =  Administrative  and  Logistics  Down  Time 
CMT  =  Corrective  Maintenance  Time 
PMT  =  Preventive  Maintenance  Time 


As  “uptime”  or  Mean  Time  Between  Maintenance  (MTBM)  increases  as  a  result 
of  increased  reliability,  operational  availability  (or  readiness)  also  increases  (DSMC, 
“Program  Managers  Tool  Kit”,  p.  43). 

2.  Impact  of  Reliability  on  Life  Cycle  Costs 

While  equipment  failure  due  to  poor  reliability  can  be  catastrophic,  leading  to  life 
or  death  implications,  reliability  of  many  products  may  be  viewed  primarily  in  economic 
terms.  Much  of  the  projected  life-cycle  cost  for  a  given  system  c  an  be  greatly  impacted 
by  decisions  made  during  the  early  stages  of  advanced  planning  and  conceptual  and 
preliminary  design.  Management  and  design  decisions  at  this  point  can  have  a  major 
impact  on  the  activities  and  operations  in  all  subsequent  phases  of  the  life  cycle.  Thus,  it 
is  critical  to  consider  reliability  and  its  affect  on  logistical  support  in  the  early  stages  of 
planning  and  design  in  an  effort  to  avoid  unplanned  excessive  O&S  costs  throughout  a 
system’s  life  cycle  and  not  postpone  reliability  considerations  to  a  downstream  activity. 
The  need  to  look  beyond  short-term  initial  cost  of  procurement  and  acquisition  and 
address  system  life-cycle  cost  is  obvious,  and  experience  has  shown  that  logistics 
requirements  can  have  a  major  impact  on  overall  life-cycle  cost  (Blanchard,  p.  4). 
Understanding  that  initial  life  cycle  supportability  requirements  to  include  integrated 
logistics  support  is  based  on  reliability  estimates,  it  becomes  clear  that  reliability  needs  to 
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be  recognized  as  a  significant  factor  throughout  the  life  cycle  while  assuming  a  major 
role  in  research,  design,  production,  and  system  performance  during  operational  use.  An 
increased  focus  on  reliability  can  lead  to  reduced  life  cycle  support  costs,  equating  to 
increased  funds  available  for  recapitalization  and  modernization  of  forces.  Likewise, 
because  of  its  recognized  importance,  it  is  mandatory  for  all  program  managers  with  the 
Department  of  Defense  to  plan  for  and  execute  measures  to  ensure  their  program 
accounts  for  the  user’s  RAM  objectives  (DoD  5000.2-R). 

Along  with  the  latest  revision  to  the  DoD  5000  series  acquisition  directives  in 
October  2000,  the  Secretary  of  Defense  issued  a  memorandum  that  outlined  six  major 
themes  in  the  updated  documents.  One  of  the  major  themes  is  that,  “The  acquisition 
process  must  consider  both  performance  requirements  and  fiscal  constraints. 
Accordingly,  cost  must  also  be  an  independent  variable  in  programmatic  decisions.”  The 
theme,  known  as.  Cost  As  an  Independent  Variable  (CATV),  is  an  initiative  intended  to 
put  focus  on  life-cycle  costs  by  considering  both  performance  requirements  and  fiscal 
constraints  by  making  cost  and  performance  trade  offs.  Over  the  past  decade,  the  relative 
importance  of  LCC  has  greatly  increased,  and  it  is  now  mandatory  for  the  major 
acquisition  category  programs.  Additionally,  many  contemporary  political  issues  dictate 
that  the  control  of  costs  associated  with  procurement  and  life  cycle  management  of 
weapon  systems  receive  an  unprecedented  level  of  management  attention  (DSMC, 
“Acquisition  Logistics  Guide”,  p.  12-1). 

The  concept  of  CAIV  must  by  utilized  in  establishing  an  effective  acquisition 
strategy.  Per  DoD  5000.2-R,  the  acquisition  strategy  shall  address  methodologies  “to 
acquire  and  operate  affordable  DoD  systems  by  setting  aggressive,  achievable  cost 
objectives  and  managing  achievement  of  these  objectives”.  A  strategy  that  considers  the 
total  cost  to  the  government  over  the  entire  cradle -to-grave  cycle  of  the  system  is 
“necessary  to  provide  balance  and  perspective  to  the  program  in  consideration  of  the 
performance  and  schedule  requirements  to  avoid  suboptimization”.  In  this  regard, 
program  managers  primary  focus  should  be  on  minimizing  life  cycle  cost  (DSMC, 
“Acquisition  Strategy  Guide”,  p.  2-12). 
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a.  Background  and  Components  ofLCC 

DOD  TOC  is  comprised  of  costs  to  research,  develop,  acquire,  own 
operate,  and  dispose  of  weapon  and  support  systems,  other  equipment  and  real  property, 
the  costs  to  recruit,  retain,  separate  and  otherwise  support  military  and  civil  ian  personnel, 
and  all  other  costs  of  business  operations  of  the  DOD.  Defense  Systems  TOC  is  defined 
as  Life  Cycle  Cost.  LCC  (per  DoD  5000.4M)  includes  not  only  acquisition  program 
direct  costs,  but  also  the  indirect  costs  attributable  to  the  acquisition  program  (i.e.,  costs 
that  would  not  occur  if  the  program  did  not  exist).  For  example,  indirect  costs  would 
include  the  infrastructure  that  plans,  manages,  and  executes  a  program  over  its  full  life 
and  common  support  items  and  systems. 

For  purposes  of  cost  estimating,  LCC  is  typically  divided  into  research  and 
development  (R&D),  procurement,  operations  and  support  (O&S),  and  disposal.  Life 
Cycle  Costs  involves  all  costs  associated  with  the  system  life  cycle,  to  include  the 
following: 

•  Research  and  development  (R&D)  cost.  Those  costs  incurred  from 
program  initiation  at  the  conceptual  through  the  end  of  engineering  and 
manufacturing  development.  R&D  costs  include  the  cost  for  feasibility 
studies,  modeling,  tradeoff  analyses,  engineering  design,  development, 
fabrication,  assembly  and  test  of  prototype  hardware  and  software,  system 
test  and  evaluation,  associated  peculiar  support  equipment,  and 
documentation. 

•  Procurement  cost.  Includes  the  costs  associated  with  producing  or 
procuring  the  prime  hardware,  support  equipment,  training,  data,  initial 
spares,  and  facilities. 

•  Operation  and  support  (O&S)  cost.  Consists  of  all  costs  incurred  by  the 
DOD  to  field/deploy  the  system  including  personnel,  consumable  and 
reparable  parts,  fuel,  shipping,  and  maintenance.  Includes  the  cost  of 
sustaining  operation,  personnel  and  maintenance  support,  spare/repair 
parts  and  related  inventories,  test  and  support  equipment  maintenance, 
transportation  and  handling,  facilities,  modifications  and  technical  data 
changes,  and  so  on. 

•  System  retirement  or  disposal  cost.  Captures  costs  associated  with 
deactivating  or  disposing  of  a  materiel  system  at  the  end  of  its  useful  life. 
(DSMC,  “Acquisition  Logistics  Guide”,  pp.  12-3  -  12-4). 
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As  depicted  by  the  categories  listed  above,  life  cycle  cost  of  a  weapon 
system  begins  with  the  determination  of  a  mission  requirement  and  continues  through 
design,  development,  production,  operation,  support,  and  eventually  the  disposal  and 
demilitarization  of  the  system  at  the  end  of  its  useful  life.  It  is  widely  accepted  within  the 
acquisition  community,  that  the  costs  of  operating  and  supporting  a  weapon  system  far 
exceed  the  actual  procurement  costs  incurred  through  the  design,  development,  and 
production  of  a  new  system.  Although  the  percentage  of  life-cycle  costs  attributable  to 
each  element  is  not  identical  for  all  weapon  systems,  there  is  little  variation  across  the 
range  of  various  systems.  The  historical  life-cycle  cost  percentage  breakdown  for  major 
defense  weapon  systems  is  depicted  in  Figure  2.2  (OSD  CAIG,  “O&S  Cost  Estimating 
Guide”). 


ENGMEElma 
MANUFACTURING 
DEVELOPMENT  PHASE 


Figure  2.2.  Weapon  System  Life  Cycle  Cost  Breakdown. 


While  production  may  be  viewed  as  the  most  costly  portion  of  the 
program  per  unit  of  time,  it  actually  only  amounts  to  roughly  30%  of  the  LCC.  Based  on 
these  figures,  it  becomes  readily  apparent  that  the  largest  cost  driver  in  the  life  of  a 
system  is  the  O&S  phase.  To  further  compound  this  figure,  when  today’s  aging  systems 
exceed  their  originally  intended  life  expectancy,  O&S  costs  can  actually  form  75  -90%  of 
a  system’s  LCC  (Parker,  p.  275).  Understanding  that  an  increasing  portion  of  the  defense 
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budget  is  being  consumed  by  growing  O&S  expenditures,  there  has  understandably  been 
considerable  effort  to  reduce  such  costs.  Ultimately,  the  increase  of  fun  ds  available  for 
recapitalization  and  modernization  of  legacy  systems  will  result  through  the  reduction  of 
O&S  funds. 

In  the  past,  total  system  cost  has  either  not  been  obvious  or  has  been 
somewhat  ignored  due  to  incentive  and  managerial  issues,  partic  ularly  those  costs 
associated  with  operation  and  support.  As  previously  discussed,  a  major  portion  of  the 
projected  life-cycle  cost  for  a  given  system  or  product  results  from  the  consequences  of 
decisions  made  during  the  early  phases  of  program  planning  and  system  conceptual 
design.  Referring  back  to  Figure  2.2,  while  the  greatest  proportion  of  life  cycle  costs 
occur  during  the  operation  and  support  phase  of  a  program,  the  greatest  opportunity  for 
influencing  these  costs  occurs  during  the  early  phases  of  the  program  as  shown  in  Figure 
2.3. 


Figure  2.3.  Commitment  of  Life-Cycle  Cost.  (From:  Blanchard,  p.  82) 

The  recent  CAIV  acquisition  reform  initiative  is  a  way  of  developing  life  - 
cycle  cost  targets  for  the  system  to  be  acquired  and  constraining  the  system  design  trade¬ 
offs  by  the  target  cost  of  system  ownership.  Prior  to  the  CAIV  concept,  the  Design -to- 
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Cost  approach  (DTC)  was  very  prominent  within  the  acquisition  community.  However, 
the  DTC  approach  had  primarily  concentrated  on  controlling  system  procurement  costs, 
rather  than  life-cycle  cost.  As  a  result,  DTC  created  the  wrong  incentives  for  former 
program  management  offices,  resulting  in  programs  that  did  not  adequately  address 
sustainment  and  life  cycle  cost  consequences  of  early  acquisition  decisions. 

b.  Life  Cycle  Cost  Analysis 

Life  cycle  cost  analysis  is  typically  part  of  the  supportability  analysis, 
discussed  later,  and  is  conducted  to  address  the  total  cost  of  a  system  and  its  supporting 
activities  throughout  its  planned  life  cycle.  Such  an  analysis  includes  the  estimation  of 
the  system  life  cycle  cost  (design  and  development,  production  and/or  construction, 
system  utilization,  maintenance  and  support,  and  retirement/disposal  costs),  high -cost 
contributors,  cause-and-effect  relationships,  potential  areas  of  risk,  and  identification  of 
areas  for  improvement  or  cost  reduction  (Blanchard,  p.  176).  Due  to  the  fact  that  much 
of  the  downstream  cost  is  the  consequence  of  design  and  management  decisions  made 
during  the  early  stages  of  conceptual  and  preliminary  design,  the  use  of  life  cycle  cost 
analysis  is  critical  if  a  program  management  office  is  to  assess  whether  or  not  the  system 
can  be  operated  and  supported  in  an  effective  and  efficient  manner  throughout  its 
intended  life  cycle. 

Many  factors  are  involved  with  the  estimation  of  life  cycle  costs. 
Specifically,  reliability  considerations,  estimates,  and  the  accuracy  of  such  estimates  play 
a  significant  role  in  LCC  estimations.  The  fundamental  objective  of  LCC  reduction 
analysis  is  to  identify  the  cost  drivers  that  most  significantly  affect  life  cycle  costs.  Such 
analyses  allow  for  trade  off  considerations  with  respect  to  different  courses  of  action. 
During  each  phase  of  the  acquisition  cycle,  engineers  and  managers  provide  prompt 
feedback  regarding  the  costs  of  new  or  alternative  designs  or  other  economical  solutions 
with  respect  to  their  effect  on  LCC  forecasts.  Likewise,  engineers  and  managers  must 
achieve  a  proper  balance  between  acquisition  decisions  and  costs  and  the  resulting 
(predicted)  operation  and  support  costs.  Figure  2.4  illustrates  the  design  linkage  with 
operation  and  support  cost  drivers  (DSMC,  “Designing  Quality  into  Defense  Systems”,  p. 
41-42). 
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Figure  2.4.  Design  and  Life  Cycle  Cost  Linkage.  (From:  DSMC,  “Designing  Quality 

into  Defense  Systems”,  p.  42) 

There  are  countless  examples  of  how  reliability  improvements  in  both 
Government  and  industry  have  resulted  in  substantial  cost  reductions.  It  is  well  know 
throughout  the  current  acquisition  community  that  initial  investments  in  the  design, 
development,  and  production  of  reliable  weapon  systems  can  have  significant  impacts  on 
reducing  O&S  costs  and  ultimately  LCC.  Such  an  example  is  the  DoD’s  Minuteman  I 
missile  system  which  implemented  a  reliability  improvement  study  that  eventually  led  to 
a  30%  reduction  in  the  failure  rate.  The  cost-effectiveness  analysis  revealed  a  return  of 
eight  dollars  for  every  dollar  invested  in  reliability  improvement.  The  net  savings  over  a 
ten-year  period  was  expected  to  be  $160,000,000  (Kececioglu,  p.  23).  Another  example 
of  the  potential  cost  savings  can  be  found  with  the  F-105  weapon  system,  which,  by  way 
of  implementing  a  reliability  improvement  program,  increased  system  reliability  from 
.7263  to  .8986.  While  the  reliability  program  nonrecurring  costs  were  estimated  at 
$25,500,000,  the  annual  savings  in  maintenance  costs  were  estimated  at  $54,000,000 
(Kececioglu,  p.  24).  It  is  clear  that  while  upfront  investments  in  reliability  may  increase 
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initial  procurement  costs,  the  significant  savings  resulting  from  the  potential  reductions  in 
O&S  and  LCC  quickly  outweigh  any  upfront  costs. 

c.  Break  Even  Analysis 

A  program  must  consider  cost  during  reliability  and  maintainability  design 
balancing  activities.  Fortunately,  Life  Cycle  Cost  models  are  available  and  often  used  as 
a  vehicle  by  which  estimates  for  operation,  performance,  reliability  and  maintainability, 
and  cost  are  traded  off  to  obtain  “design  to”  target  goals  which  collectively  represent  a 
balanced  design.  For  the  purpose  of  considering  cost  trade-offs,  additional  relationships 
are  developed  which  define  how  cost  changes  as  reliability  and  maintainability  is  varied 
from  a  baseline.  Specifically,  as  a  system  is  made  more  reliable,  the  operating  cost 
should  decrease  since  there  are  fewer  failures  to  repair.  At  the  same  time,  it  is  anticipated 
that  acquisition  cost  (development  and  production)  will  increase  to  attain  higher 
reliability  in  the  system  (DSMC,  “Designing  Quality  into  . . .”,  p.  11). 

As  discussed,  improvements  in  system  reliability,  to  a  feasible  extent, 
dramatically  decrease  system  LCC.  However,  increasing  system  reliability  beyond 
feasible  technological  levels  may  require  an  enormous  amount  of  resources  to  be 
consumed  during  research  and  development  (R&D)  to  the  point  that  the  cost  savings 
from  improved  reliability  may  not  offset  such  costs,  resulting  in  less  than  optimal  LCC. 
The  theoretical  relationship  between  system  reliability  and  LCC  is  depicted  in  Fig  ure  2.5. 


Figure  2.5.  Reliability  and  LCC  Tradeoff. 
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For  proper  economic  analysis,  one  must  consider  the  costs  associated  with 
the  entire  life  cycle  of  a  system,  evaluating  the  trade-off  between  increased  early 
investments  in  reliability  improvement  and  the  resulting  future  cost  savings.  When 
comparing  alternatives,  a  program  management  office  must  consider  both  the  aspects  of 
cost  effectiveness  and  the  point  in  time  where  one  alternative  becomes  more  cost  - 
effective  than  another  alternative.  A  break-even  analysis  is  an  approach  where  the 
cumulative  costs  for  two  or  more  investment  alternatives  (or  programs)  are  estimated, 
projected,  and  compared  with  respect  to  time.  In  the  event  that  the  break-even  point  is 
realistic  in  terms  of  expected  system  life,  then  it  may  cost-effective  to  consider  the 
increased  early  investment  during  Research  and  Development  phases  in  order  to  achieve 
higher  system  reliability.  Figure  2.6  provides  a  comparison  of  two  alternatives  where  it 
appears  that  the  increased  investment  during  R&D  results  in  a  more  cost-effective  option 
in  the  long  run. 


Figure  2.6.  Break-Even  Analysis.  (From:  Blanchard,  p.  89) 
cl.  Cost  Effectiveness 

It  is  important  to  understand  that  when  decidin  g  upon  the  optimal  level  of 
reliability  to  be  designed,  manufactured,  and  maintained  into  a  product,  it  is  not 
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necessarily  the  point  at  which  the  cost  to  own,  operate,  and  maintain  the  product  for  its 
desired  life  is  minimum.  Rather,  the  primary  objective  should  be  to  develop  a  system 
that  is  most  cost-effective,  within  the  constraints  of  operational  and  maintenance 
requirements.  In  other  words,  the  acquisition  community  should  not  aim  to  strictly 
minimize  LCC,  and  instead,  should  consider  cost  effectiveness  as  it  relates  to  the  measure 
of  a  system  in  terms  of  mission  fulfillment  (system  effectiveness)  and  total  life  cycle 
costs  (Blanchard,  p.  34).  Cost  effectiveness  involves  a  cost -benefit  analysis  factor 
employed  for  decision-making  purposes. 

When  considering  cost  effectiveness,  the  aspects  of  effectiveness  must  be 
quantified  and  depend  upon  the  specific  mission  or  system  characteristic  that  a  program 
desires  to  specify  and  measure.  While  measuring  effectiveness,  one  must  consider: 

•  System  performance  aid  physical  parameters:  capacity,  delivery  rate, 
power  output,  range,  accuracy,  volume,  speed,  weight,  etc. 

•  System  operational  and  support  factors:  availability,  dependability, 
capability,  operational  readiness,  reliability,  maintainability,  etc. 

•  Total  Ufe-cycle  cost:  research  and  development,  production/construction 
cost,  operation  and  maintenance  cost,  retirement  and  disposal  cost 
(Blanchard,  p.  83) 

In  order  to  achieve  a  desirable  cost  effectiveness,  a  relationships  must  be 
established  between  performance  and  operational  parameters  and  cost.  Figure  2.7 
illustrates  an  example  of  the  relationship  between  reliability  (MTBF)  and  total  life  cycle 
cost,  where  the  objective  is  to  design  a  design  a  weapon  system  that  meets  a  specified 
reliability  level  within  a  given  budget  level  and  yet  be  most  cost-effective.  System 
design  characteristics  are  evaluated  in  terms  of  reliability  and  cost,  and  as  a  result,  design 
changes  are  recommended  in  an  effort  to  achieve  the  point  on  the  curve  near  the 
minimum  cost  (Blanchard,  p.  88). 
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Figure  2.7.  Reliability  versus  Cost.  (From:  Blanchard,  p.  88) 

There  is  a  significant  increase  in  costs  associated  with  achieving  higher 
levels  of  reliability.  In  fact,  the  marginal  increase  in  reliability  bee  omes  increasingly 
smaller  and  the  marginal  cost  becomes  increasingly  larger  as  developers  attempt  to 
maximize  the  level  of  reliability.  In  other  words,  in  may  be  relatively  inexpensive  to 
increase  reliability  from  50%  to  70%  while  it  may  be  far  more  costly  to  increase  system 
reliability  from  98%  to  99%.  Therefore,  it  is  not  typically  optimal  to  strive  for  100% 
reliability.  Figure  2.8  illustrates  the  diminishing  marginal  gain  associated  with  achieving 
higher  levels  of  reliability. 
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Level  of  Reliability 


Figure  2.8.  Reliability  S-Curve.  (NPS  Logistics  Engineering  Class  Notes) 

e.  Life  Cycle  Cost  Models 

Numerous  commercial  life  cycle  cost  models  have  been  developed  in  an 
effort  to  help  Program  Managers  structure  and  analyze  large  amounts  of  data  used  to 
support  major  LCC  decisions.  One  of  the  major  advantages  of  the  LCC  models  is  their 
ability  to  provide  early  input  to  the  front-end  design  analysis  stage  of  the  Concurrent 
Engineering  (CE)  and  Logistic  Supportability  (LS)  processes.  Basically,  the  models 
available  are  database  managers  that  have  the  capability,  to  varying  degrees,  to  import, 
modify,  analyze,  integrate,  and  manage  large  amounts  of  data  from  many  different 
sources.  Reports  can  be  generated  that  display  or  project  the  overall  effects  and  results  of 
program  decisions  on  existing  or  alternative  system  designs,  including  risks  thereof  while 
storing  a  baseline  of  program  decisions.  The  life  cycle  cost  models  provide  a  design  and 
support  system  tradeoff  with  sensitivity  and  comparative  analyses,  providing  the 
flexibility  of  rapidly  assessing  the  reliability,  LCC  and  logistic  supportability  impacts  of 
various  equipment  configurations  and  other  design  supportability  options  (Sterling, 
“Analysis  of  LCC  Models  for  DoD”).  Some  of  the  life  cycle  cost  models  available  to 
Program  Management  Offices  include  but  are  not  limited  to  EDCAS,  ACEIT,  FLEX+, 
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CASA,  and  the  COCOMO  model,  all  of  which  offer  varying  degrees  of  advantages  as 
well  as  disadvantages  relative  to  the  others.  The  specific  application  of  the  models  will 
not  be  discussed  in  this  text  as  it  is  beyond  the  scope  of  the  thesis. 

D.  SUPPORTABILITY  ANALYSIS 

Supportability  analyses  are  a  wide  range  of  related  analyses  that  should  be 
conducted  within  the  systems  engineering  process.  Specifically,  supportability  analysis 
(SA)  is 

...  an  iterative  analytical  process  by  which  the  logistic  support  necessary 
for  a  new  (or  modified)  system  is  identified  and  evaluated.  The  SA 
constitutes  the  application  of  selected  quantitative  methods  to  (1)  aid  in 
the  initial  determination  and  establishment  of  supportability  criteria  as  an 
input  to  design;  (2)  aid  in  the  evaluation  of  various  design  alternatives; 

(3)  aid  in  the  identification,  provisioning,  and  procurement  of  the  various 
elements  of  maintenance  and  support;  and  (4)  aid  in  the  final  assessment 
of  the  system  support  infrastructure  throughout  the  utilization  phase 
(Blanchard,  p.  24). 

Reliability  characteristics  inherent  within  the  system  design  actually  dictate  the 
requirements  for  the  subsequent  maintenance  and  support  of  that  system  throughout  its 
lifecycle,  and  thus,  program  offices  must  establish  the  appropriate  logistic  support 
requirements  in  the  early  stages  of  conceptual  design  (Blanchard,  p.  252).  However,  in 
addition  to  actual  inherent  reliability  associated  with  system  design,  under-  or  over- 
estimations  of  the  reliability  of  weapon  systems  in  development  can  dramatically,  and 
often  adversely,  affect  life  cycle  cost  and  operational  availability  as  the  reliability 
estimate  provides  the  basis  for  initial  life  cycle  supportability  decisions.  Therefore, 
accurate  reliability  predictions  and  thorough  analyses  are  required  as  an  integral  input  to 
the  supportability  analysis. 

The  supportability  analysis  includes  two  major  areas  of  focus.  The  firs  t  is  the 
accomplishment  of  design  trade-off  studies,  level  of  repair  analyses,  life-cycle  cost 
analyses,  and  related  activities  directed  toward  the  objective  of  designing  for 
supportability.  The  second  area  of  focus  involves  the  evaluations  of  the  system  design 
configuration,  as  it  exists  at  the  time,  with  the  objective  of  defining  logistic  support 
resource  requirements  (i.e.,  spare/repair  parts,  test  and  support  equipment,  number  of 
maintenance  personnel,  level  of  personnel  training,  etc.).  With  the  identification  of 
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specific  logistics  requirements  identified,  the  provisioning,  procurement,  and  acquisition 
process  commences  (Blanchard,  p.  355).  Ultimately,  the  supportability  analysis  leads  to 
a  database  that  assists  in  identifying  the  specific  requirements  leading  to  the  development 
of  the  maintenance  and  support  infrastructure.  The  overall  intent  is  to  design  or  develop 
a  system  that  will  meet  the  specified  operational  requirements  in  an  effective  and 
efficient  manner  by  maximizing  system  effectiveness  while  minimizing  life  cycle  cost. 

E  RELIABILITY  ANALYSIS  AND  AVAILABLE  TO  OLS 

The  reliability  analyses  can  be  used  to  define  the  quantitative  parameters  for  a 
system,  subsystem,  or  component,  and  it  is  often  expressed  in  number  of  failures  in  a 
given  set  period  of  time,  set  number  of  cycles,  or  a  set  number  of  operations  (i.e.,  rounds 
fired,  number  of  starts,  etc).  As  engineering  data  become  available,  reliability  prediction 
serves  as  a  check  on  the  design  in  relation  to  the  system  requirement,  indicating  areas  of 
incompatibility  that  may  need  evaluated  for  design  improvement.  As  previously 
discussed,  the  level  of  reliability  achieved  in  fielded  systems  directly  affects  operational 
availability  and  sustainment  requirements.  Therefore,  successful  system  de  signs  require 
that  component  and  system  reliability  be  predictable.  This  requires  that  a  reliability 
program  be  established  to  assess  the  reliability  of  system  components.  Accurate  data  is 
crucial  in  establishing  reliability  information,  and  the  more  dat  a  available,  the  greater  the 
confidence  that  can  be  expressed  in  the  estimated  or  predicted  reliability  level. 

During  logistical  support  planning,  the  Marine  Corps  is  forced  to  rely  upon 
estimates,  and  unfortunately,  reliability  data  is  often  difficult  to  obtain,  as  it  is  acquired 
through  observing  the  failure  of  products  and  their  components.  This  requires  life 
testing,  in  which  a  number  of  items  are  tested  until  a  significant  number  of  failures  occur. 
However,  such  tests  are  often  very  expensive,  since  they  are  destructive,  and  to  obtain 
meaningful  statistics,  substantial  numbers  of  the  system  or  subsystem  must  fail.  The  tests 
are  also  time  consuming,  since  few  unbiased  acceleration  methods  are  available  to  greatly 
compress  the  time  to  failure,  the  test  time  may  be  comparable  or  longer  than  the  normal 
product  life.  Reliability  data  is  also  collected  from  field  failures  once  a  product  is  put 
into  operational  use.  However,  this  is  a  lagging  indicator  and  is  not  nearly  as  useful  as 
results  obtained  earlier  in  the  development  process  (Lewis,  p.  49).  Additionally,  it  is 
important  that  reliability  be  considered  in  the  concept  and  design  process  because 
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identifying  and  correcting  related  problems  in  later  stages  of  the  life  cycle  has  an  adverse 
cost  leverage  as  shown  in  Figure  2.9. 


Program  Phases 


Figure  2.9.  Relative  Costs  of  Problem  Correction  versus  Program  Phase.  (DSMC, 
“Designing  Quality  into  .  .  .”  p.  30) 

Multiple  potential  opportunities  are  present  throughout  the  acquisition  life  cycle 
to  address  reliability.  Beginning  with  the  initial  requirements  generation,  through  each 
iteration  of  the  systems  engineering  process,  and  ultimately  during  post -production, 
reliability  must  be  planned  for,  monitored,  accessed,  and  improved  during  the  matu  ration 
of  a  weapon  system  (Ryan,  p.  1).  The  program’s  application  of  special  reliability  tasks 
enhances  the  capability  of  satisfying  the  warfighter  or  user’s  needs.  Flowever,  reliability 
tasks  must  be  fully  integrated  into  the  total  technical  program  and  be  performed 
concurrent  with  other  engineering  tasks  to  ensure  reliability  is  designed  -in  before  design 
maturity  reaches  a  stage  when  engineering  changes  become  costly  to  implement 
(Reliability,  Maintainability,  and  Supportability  Guidebook,  SAE  Internatio  nal,  p.  70). 
While  the  list  of  key  reliability  tasks,  below,  serve  slightly  different  purposes,  they  are 
applicable  to  varying  equipment  types,  and  range  in  depth,  scope,  and  complexity  of  the 

task,  if  properly  conducted,  all,  in  some  capacity,  can  provide  a  valuable  contribution  to 
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the  design  and  development  of  a  system  with  respect  to  reliability  performance.  These 
tasks  must  be  tailored  to  fit  the  particular  program  need.  Furthermore,  the  tasks  listed  are 
only  those  with  the  widest  acceptance  and  application  within  program  management  and 
are  not  all-inclusive. 

•  Reliability  Requirements  Definition 

•  Reliability  Program  Plan 

•  Reliability  Design  Standards/Guides/Checklists 

•  Environmental  Criteria 

•  Reliability  Modeling 

•  Reliability  Allocation  and  Apportionment 

•  Reliability  Prediction 

•  Subcontractor/Supplier  Monitoring  and  Control 

•  Reliability  Design  Evaluation 

•  Failure  Modes  Effects  and  Criticality  Analysis  (FMECAS) 

•  Process  Failure  Modes  and  Effects  Analysis  (PFMEA) 

•  Reliability  Development/Growth  Test  (RD/GT) 

•  Weibull  Analysis 

•  Failure  Reporting,  Analysis,  and  Corrective  Action  (FRACAS) 

•  Software  Reliability  Assessment 

•  Parts  Control  Program 

•  Environmental  Stress  Screening  (ESS) 

•  Reliability  Qualification  Test  (RQT)  Program 

•  Probabilistic  Design  Assessment  for  Reliability 

•  Fault  Tree  Analysis 

•  Part  Stress  Derating 

•  Worst  Case  Circuit  Analysis 

The  integrated  analyses  can  include  any  number  of  tools,  practices,  or  techniques 
to  realize  reliability  and  supportability  characteristics.  The  tasks  above,  or  some 
combination  of  them,  should  be  selectively  applied  to  each  program  based  on  the 
program’s  life  cycle,  system  complexity  and  type,  technology  advancement,  and  schedule 
and  cost  constraints.  If  the  selected  reliability  tasks  are  appropriately  tailored  for  scope 
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and  depth  and  adequately  integrated  with  other  program  tasks,  an  effective  reliability 
program  will  result  ((Reliability,  Maintainability,  and  Supportability  Guidebook,  SAE 
International,  p.  71). 

F.  PROGRAM  MANAGER/WEAPON  SYSTEM  MANAGEMENT 
IMPLEMENTATION 

Program  Management/Weapon  Systems  Management  (PMAVSM)  is  defined  as 
“the  planning,  organizing,  acquisition,  controlling,  sustainment,  and  disposal  of  weapon 
systems  and  secondary  items  in  support  of  validated  Marine  Corps  requirements,”  while 
Supply  Chain  Management  is  defined  as  “the  planning,  organizing,  and  controlling  of 
supply  chain  activities  for  the  Marine  Crops  wholesale  and  retail  supply  business  to 
maintain  and  support  assigned  principle  end  items  and  secondary  items”  (PMAVSM 
“Activities  Definitions”).  Under  the  recent  PMAVSM  initiative,  traditional  roles, 
responsibilities,  resources,  and  billets  of  Marine  Corps  Systems  Command 
(MARCORSYSCOM)  and  Marine  Corps  Logistics  Base  (MARCORLOGBASES)  were 
realigned  to  optimize  Life  Cycle  Management  of  weapon  systems.  The  initiative  was 
established  to  “clearly  delineate  authority,  responsibility,  and  accountability  of  managers 
and  organizations”  (Williams,  PMAVSM  Slide  Show  dtd  17  Jan  01). 

Prior  to  the  Program  ManagerAVeapon  System  Manager  Implementation  efforts,  a 
major  weapon  system  was  procured  and  fielded  at  MARCORSYSCOM  and  was  passed 
on  to  MARCORLOGBASES  for  Sutainment/Life  Cycle  Management.  As  a  result  of  this 
disjointed  process,  major  weapon  systems  entered  the  Lleet  and  encountered  severe 
readiness  and  supportability  problems  (MARCORSYSCOM  Study  Plan  “Sustainment 
Consequences  .  .  .”,  p.  1).  It  has  been  argued  that  prior  to  the  implementation  of 
PMAVSM,  the  incentives  in  place  for  program  managers  caused  them  to  focus  on  short  - 
term  program  objectives  that  they  were  evaluated  on  such  as  procurement  cost,  schedule, 
and  performance.  Additionally,  few  if  any,  incentives  were  in  place  that  encouraged 
program  mangers  to  analyze  long-term  sustainment  and  life  cycle  cost  consequences  of 
their  early  acquisition  decisions.  However,  under  the  realignment  of  responsibilities 
within  Materiel  Command  (MATCOM),  LOGBASES,  and  MARCORSYSCOM,  Marine 
Corps  Systems  Command  became  responsible  for  the  availability  of  equipment  through 
the  entire  materiel  life  cycle.  As  a  result,  the  decisions  made  early  in  the  life  cycle  of  a 
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system,  that  often  have  a  tremendous  impact  on  availability  and  sustainment,  will  directly 
impact  the  program  management  offices  throughout  the  life  span  of  the  respective 
weapon  systems. 

G.  CHAPTER  SUMMARY 

This  chapter  established  the  definite  relationships  between  reliability,  logistics, 
life-cycle  support  costs,  and  operational  availability.  In  doing  so,  the  researcher 
illustrated  the  fact  that  the  reliability  of  a  weapon  system  directly  impacts  the  operational 
availability  and  the  life  cycle  cost  of  the  system,  making  it  of  fundamental  importance  to 
PM,  logisticians,  and  warfighters  alike.  Appropriately,  the  core  of  logistical  support 
planning  focuses  on  reliability,  in  an  attempt  to  ensure  that  warfighters  are  provided  with 
capable,  supportable,  and  cost  effective  weapon  systems  that  enable  them  to  successfully 
complete  the  mission  on  the  battlefield. 

Chapter  III  will  provide  an  overview  of  the  acquisition  process  while  providing 
specific  reference  to  opportunities  within  systems’  life  cycles  for  program  managers  to 
address  reliability. 
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III.  BACKGROUND  AND  OVERVIEW  OF  RELIABILITY 
WITHIN  THE  ACQUISITION  PROCESS 


Reducing  the  cost  to  acquire  and  operate  the  department’s  equipment 
while  maintaining  a  high  level  of  performance  for  the  user  is  my  highest 
priority.  -  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
memorandum  dated  04  December  1995 

A.  INTRODUCTION 

One  of  the  first  major  steps  in  the  development  of  reliability  focus  in  DoD 
acquisitions  came  in  July  1980,  when  the  DoD  indicated  an  emphasis  on  reliability  and 
maintainability  by  publishing  a  policy  directive  on  the  subject  in  the  form  of  DoDD 
5000.40.  Until  recently,  there  has  been  a  lack  of  management  emphasis  on  the  support 
engineering  disciplines  such  as  reliability,  and  thus,  the  timely  application  of  engineering 
techniques  had  not  always  been  practiced.  As  a  result,  the  efforts  were  not  as  supportable 
and  cost  effective  as  they  could  have  been.  Today,  with  the  high  level  of  TOC  interest  in 
the  DoD,  the  management  attention  and  interest  is  present,  and  as  a  result,  we  continue  to 
make  advancements  in  the  way  of  reliability -focused  acquisitions  (Reliability, 
Maintainability,  and  Supportability  Guidebook,  SAE,  p.  64). 

This  chapter  provides  the  reader  with  background  information  on  the  defense 
acquisition  process  and  serves  to  establish  an  understanding  of  general  opportunities  for 
reliability  management  within  the  process.  First,  an  examination  of  current  DoD  and 
Marine  Corps  policies,  regulations,  and  guidance  is  provided  to  establish  the  basis  within 
which  the  acquisition  community  must  operate  to  manage  reliability  within  a  program. 
Next,  an  overview  of  the  acquisition  process  is  provided,  highlighting  opportunities  for 
reliability  management  throughout  the  process.  Finally,  the  chapter  will  conclude  by 
examining  the  existing  roles,  metrics,  and  incentives  that  guide  the  various  organizations 
and  individuals  involved  in  the  acquisition  process. 

B.  DOD  AND  MARINE  CORPS  POLICIES,  REGULATIONS,  AND 
GUIDANCE  ON  RELIABILITY 

Past  and  present  Administrations  and  Congresses  have  instituted  many  initiatives 
to  improve  the  acquisition  of  defense  systems.  In  particular,  the  publication  of  the  DoD 
5000  Series  of  Directives  in  February  1991  resulted  from  the  culmination  of  a 
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cooperative  effort  within  the  DoD  to  streamline  policy  by  standardizing  acquisition 
procedures.  As  a  result,  all  acquisition  tasks  that  were  common  among  the  service 
components  were  combined  into  the  top-level  policy,  resulting  in  the  cancellation  of  65 
other  directives.  The  Department  of  the  Navy  implemented  the  DoD  directives  in 
SECNAVINST  5000.2A  in  December  1992,  resulting  in  the  cancellation  of  39  additional 
directives.  The  Marine  Corps  implementation  of  the  SECNAV  policy  in  May  1994 
resulted  in  the  cancellation  of  14  additional  policy  directives.  “The  resulting  product  of 
these  three  efforts  is  a  single  policy  source  outlining  broad  acquisition  procedures  for 
Marine  Corps  acquisition  programs”  (USMC  PM  Acquisition  Procedures  Handbook,  p. 
1-1) 

Additionally,  1996  was  a  noteworthy  year  for  acquisition  policy  changes. 
Defense  policies  now  included  acquisition  streamlining,  integrated  product  development, 
performance  specifications,  and  the  prohibition  of  most  military  specifications  and 
standards.  The  15  March  1996  reissuance  of  DoDD  5000.1  and  DoD  5000.2-R  (later 
with  change  1  of  13  December  1996)  promulgated  these  policy  changes  in  directive 
format.  The  major  focus  of  the  new  policies  are  teamwork  (IPTs),  teamwork  with 
industry,  tailoring  empowerment,  only  performing  value -adding  tasks,  employing  Cost 
As  an  independent  Variable  (CAIV),  a  preference  for  commercial  items,  and  use  of  best 
practices  (DSMC,  “Acquisition  Logistics  Guide”,  p.  1-2). 

There  are  many  sources  of  reliability  guidance  to  assist  program  offices  with 
achieving  reliability  requirements,  a  few  of  which  are  mandatory  while  others  are 
discretionary  or  even  cancelled.  In  fact,  upon  searching  the  Defense  Acquisition 
Deskbook  website  for  DoD  (discretionary  or  mandatory)  documents  containing  the  word 
“reliability,”  213  documents  were  located.  Such  policy,  regulations,  and  guidance  have 
been  established  to  emphasize  the  importance  of  reliability  and  to  ensure  that  the 
acquisition  community  is  striving  toward  improved  reliability  techniques.  As  previously 
mentioned,  much  of  the  guidance  is  very  broad  scoped,  providing  little  detail  as  to 
specific  reliability  actions  to  be  taken  in  the  acquisition  process.  Additionally,  the 
amount  of  mandatory  guidance  is  minimal  and  has  further  decreased  in  recent  years  due 
to  acquisition  reform  initiatives.  This  section  serves  to  provide  a  general  overview  of 

some  of  the  sources  of  guidance  as  well  as  the  nature  of  the  guidance. 
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1.  Mandatory  Guidance 

DoD  5000. 2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition  Programs, 
states  that  as  paid  of  the  acquisition  strategy  for  a  given  program,  program  Managers  shall 
develop  and  document  a  support  strategy  for  life-cycle  sustainment  and  continuous 
improvement  of  product  affordability,  reliability,  and  supportability,  while  sustaining 
readiness.  RAM  activities  addressed  in  DoD  5000.2-R  are  summarized  below: 

•  The  PM  shall  establish  RAM  activities  early  in  the  acquisition  cycle 

•  The  PM  shall  develop  RAM  system  requirements  based  on  the 
Operational  Requirements  Document  (ORD)  and  Total  Ownership  Cost 
(TOC)  considerations,  and  then  state  them  in  quantifiable,  operational 
terms  that  are  measurable  during  development  and  operational  testing 

•  Reliability  requirements  shall  address  mission  reliability  and  logistics 
reliability 

•  Availability  requirements  shall  address  the  readiness  of  the  system 

•  Maintainability  requirements  shall  address  servicing,  preventive,  and 
corrective  maintenance 

•  The  PM  shall  plan  and  execute  RAM  design,  manufacturing  development, 
and  test  activities  so  that  the  system  elements,  including  software,  used  to 
demonstrate  system  performance  before  the  production  decision  reflect  the 
mature  design  (DoD  5000.2-R) 

DoD  5000.1  is  another  source  of  mandatory  guidance  which  directs  that: 

Acquisition  program  managers  shall  focus  on  logistics  considerations 
early  in  the  design  process  to  ensure  that  they  deliver  reliable  systems  that 
can  be  cost-effectively  support  and  provide  users  with  the  necessary 
support  infrastructure  to  meet  peacetime  and  wartime  readiness 
requirements  (DoD  5000. 1 ) 

Lastly,  SECNAVINST  5000.2B,  Section  4.3.6  -  Reliability,  Maintainability,  and 
Availability  -  serves  to  interleave  the  higher-level  policy. 

2.  Discretionary  Guidance 

In  addition  to  the  limited  mandatory  guidance  on  reliability,  there  is  an  abundance 
of  discretionary  guidance,  consisting  mostly  of  Military  Handbooks.  Such  discretionary 
guidance  most  typically  emphasizes  integration  of  reliability  into  the  design, 
manufacturing,  and  support  process  while  providing  recommended  tools  and  procedures 
for  doing  so.  It  is  important  to  note  that  because  the  handbooks  serve  as  guidance  only. 
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they  cannot  be  cited  as  requirements.  Due  to  amount  of  existing  documents,  only  the 
most  relevant  sources  will  be  identified  in  this  section. 

Military  Handbook  (MIL-HDBK)-781A,  Handbook  for  Reliability  Test  Methods, 
Plans,  and  Environments  for  Engineering,  Development  Qualification,  and  Production  , 
provides  a  list  of  reliability  test  methods,  reliability  test  plans,  and  envir  onmental  profile 
data  that  can  be  used  as  a  guide  when  testing  systems  for  contractual  reliability 
requirements  during  developmental  testing. 

MIL-HDBK-189,  Reliability  Growth  Modeling,  outlines  reliability  growth 
concepts  and  methodologies  for  management  of  reliability  growth  during  the 
developmental  stage  by  presenting  fundamental  concepts  followed  by  details  for  concept 
implementation. 

MIL-HDBK-502,  Acquisition  Logistics,  offers  guidance  on  acquisition  logistics  as 
an  integral  part  of  the  systems  engineering  process,  to  include  technical  and  management 
activities  associated  with  the  design,  development,  test,  production,  fielding,  sustainment, 
and  improvement  modifications.  Additionally,  the  handbook  offers  methods  to  “identify, 
consider,  and  trade-off  support  considerations  with  other  system  cost,  schedule,  and 
performance  elements  to  arrive  at  an  optimum  balance  of  system  requirements  that  meet 
the  user’s  operational  and  readiness  requirements”  (MIL-HDBK-502,  Section  4). 

The  “US  Marine  Corps  Program  Managers  Acquisition  Procedures  Handbook” 
implements  DoD,  DON,  and  Marine  Corps  directives  on  Defense  Systems  Acquisition. 
Additionally,  the  handbook  serves  as  a  “summation  of  Marine  Corps  and,  if  appropriate, 
MARCORSYSCOM  philosophy  and  policy  regarding  selected  acquisition  subject  areas” 
(USMC  PM  Acquisition  Procedures  Handbook,  p.  ii).  However,  the  handbook  offers 
minimal  guidance  concerning  reliability  related  actions  to  be  taken  during  the  respective 
phases  of  the  acquisition  process. 

Lastly,  the  DoD  Defense  Acquisition  University  (DAU)  has  published  a  series  of 
guidebooks  that  are  utilized  during  their  courses  of  acquisitions  instruction  at  Fort 
Belvoir,  Virginia.  While  designed  to  be  technical  management  educational  guides 
written  from  a  DoD  perspective,  the  guidebooks  reflect  the  latest  DoD  acquisition 
policies  and  procedures  as  described  in  the  5000  series. 
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DoD-  and  Marine  Corps -specific  policy,  regulation,  and  guidance  on  reliability 
exist  to  establish  the  basis  within  which  the  acquisition  community  should  operate  to 
manage  reliability  within  a  program.  While  there  is  an  abundance  of  DOD 
documentation  concerning  reliability  within  the  acquisition  process,  most  is  discretionary 
with  little  mandatory  guidance  and  procedures  on  the  subject.  Addit  ionally,  what  is  in 
print  is  often  very  vague  in  nature  and  provides  little  specific  guidance  to  the  Program 
Mangers. 

C.  OVERVIEW  OF  THE  ACQUISITION  PROCESS 

The  Program  Manager  must  consider  reliability  and  other  acquisition  logistics 
management  activities  throughout  the  system  development  to  ensure  the  design  and 
acquisition  of  cost-effective,  supportable  systems  and  to  ensure  that  these  systems  are 
provided  to  the  warfighter  with  the  necessary  support  infrastructure  for  achieving  the 
user’s  peacetime  and  wartime  readiness  requirements  (DSMC,  “Acquisition  Logistics 
Guide”,  p.3-11).  Consequently,  logistics  requirements  must  be  initially  planned  from  the 
beginning,  and  subsequently  into  the  system  design  process.  Reliability  tasks  must  be 
fully  integrated  into  the  program  and  be  performed  concurrent  with  other  engineering 
tasks  to  insure  reliability  is  designed -in  before  design  maturity  reaches  a  stage  when 
changes  become  costly  to  implement.  In  the  past,  the  emphasis  on  delivering  capability 
(performance)  in  a  timely  manner  (schedule)  within  procurement  cost  objectives  has 
often  overridden  reliability  and  total  ownership  cost  considerations.  Likewise,  logistics 
has  been  considered  as  a  “bill  to  be  paid  later,”  and  thus,  DoD  often  struggles  to 
efficiently  and  effectively  maintain  its  existing  mature  weapon  systems  on  today’s 
battlefields. 

In  the  defense  sector,  there  has  been  a  recent  emphasis  on  early  logistical  planning 
during  the  acquisition  process  that  has  evolved  through  the  concept  of  integrated  logistic 
support  (ILC),  defined  as  a: 

Disciplined,  unified,  and  iterative  approach  to  the  management  and 
technical  activities  necessary  to  (1)  integrate  support  considerations  into 
system  and  equipment  design;  (2)  develop  support  requirements  that  are 
related  consistently  to  readiness  objectives,  to  design,  and  to  each  other; 

(3)  acquire  the  required  support;  and  (4)  provide  the  required  support 
during  the  operational  phase  at  minimum  cost.  (DSMC,  “Integrated 
Logistics  Support  Guide”) 
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As  a  result  of  the  recent  focus  on  post  deployment  logistical  supportability,  there 
has  been  an  increased  emphasis  on  the  early  opportunities  for  addressing  reliability 
within  weapon  systems  acquisition.  Initially,  the  Requirements  Generation  Process  can 
serve  as  a  primary  tool  for  the  Marine  Corps  to  document  quantifiable  system  reliability 
requirements  in  the  Operational  Requirements  Document  (ORD)  in  the  form  of  Key 
Performance  Parameters  (KPP).  The  reliability  requirements  can  be  used  in  source 
selection  as  DoD  converts  specific  performance  specifications  into  contractual  terms, 
which  should  perhaps  include  an  inherent  reliability  goal.  The  Systems  Engineering 
Process  allows  the  contractor  to  build  to  required  reliability  performance  specifications. 
Once  contractors  submit  their  reliability  estimates,  program  planning  and  organizational 
management  can  emphasize  an  independent  and  rigorous  reliability  testing  process 
throughout  the  development  phase  in  order  to  demonstrate  the  required  reliability 
performance  levels  to  ensure  the  system  will  operate  in  the  field  as  intended.  While  not 
an  upfront  opportunity,  comparison  and  assessment  of  achieved  field  reliability  to 
contractor  reliability  estimates  could  be  conducted  throughout  weapon  system  maturation 
to  ensure  attainment  of  system  reliability  as  planned. 

The  subsequent  sections  will  provide  an  overview  of  the  participants  involved  in 
the  acquisition  process,  a  summary  of  the  process  itself,  and  the  opportunities  to  address 
reliability  throughout  the  process. 

1.  Organizations  and  Participants  in  the  Marine  Corps  Acquisition 
Process 

Weapons  systems  acquisition  is  a  very  complex  process,  involving  many  different 
participants  at  varying  levels.  This  section,  a  summation  taken  from  the  “USMC 
Program  Managers  Acquisition  Procedures  Handbook,”  provides  an  overview  of  the 
organizations  and  participants  involved  as  well  as  a  brief  summary  of  their  respective 
roles. 

Before  proceeding,  it  is  important  to  note  that  the  organizational  chain  of 
command  is  not  the  same  as  the  systems  acquisition  chain  and  that  certain  levels  are 
responsible  for  requirements  while  others  are  responsible  for  implementing  those 
requirements. 
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The  chain  of  authority  for  Marine  Corps  systems  acquisition  starts  at  the 
Department  of  Defense  level  where  the  responsibility  for  acquisition  policy  and  major 
program  decision  authority  has  been  placed  with  the  Under  Secretary  of  Defense, 
Acquisition,  Technology,  and  Logistics  (USD,  (AT&L)).  The  position  of  USD  (AT&L) 
is  subordinate  only  to  the  Secretary  and  Deputy  Secretary  of  Defense.  In  the  systems 
acquisition  hierarchy,  the  USD  (AT&L)  is  the  Defense  Acquisition  Executive  (DAE), 
acting  as  the  ultimate  program  decision  authority  on  certain  major  programs  preparing  to 
move  from  one  Milestone  to  the  next. 

Immediately  below  the  USD  (AT&L)  in  the  systems  acquisition  hierarchy  is  the 
position  of  the  Assistant  Secretary  of  the  Navy,  Research,  Development,  and  Acquisition 
(ASN,  RDA).  The  ASN  (RDA)  performs  the  same  role  for  the  Secretary  of  the  Navy  that 
the  USD  (AT  &L)  does  for  the  Secretary  of  Defense.  ASN  (RDA)  is  the  sole  decision 
authority  within  the  Department  of  the  Navy  (DoN)  for  major  Navy/Marine  Corps 
programs,  and  is  responsible  for  Navy  acquisition  policy.  ASN(RDA)  also  serves  as  the 
Component  Acquisition  Executive  (CAE)  for  the  Navy,  and  is  referred  to  as  the  Navy 
Acquisition  Executive  (NAE). 

The  next  position  in  the  Marine  Corps  acquisition  hierarchy  is  the  Commandant 
of  the  Marine  Corps  (CMC).  The  CMC  is  responsible  for  determining  requirements  and 
ensuring  the  resources  (funding  and  people)  for  those  requirements.  However,  the  CMC 
is  not  directly  involved  in  the  program  decision  process.  Instead,  the  CMC  appoints  a 
Milestone  Decision  Authority  (MDA)  to  act  in  his  behalf  in  the  acquisition 
decision/policy  process,  similar  to  the  roles  performed  by  the  NAE  and  USD  (AT&L). 
The  Commander,  Marine  Corps  Systems  Command  (COMMARCORSYSCOM) 
performs  the  MDA  role  for  the  Marine  Corps.  Before  proceeding,  we  must  distinguish 
between  Marine  Corps  Systems  Command  and  the  Marine  Corps  Combat  Development 
Command  (MCCDC). 

There  are  two  major  functions  involved  in  systems  acquisition  -  requirements 
determination  and  acquisition.  As  previously  discussed,  the  CMC’s  role  at  the  top  level 
is  primarily  with  requirements  determination.  However,  the  Commanding  General, 
MCCDC  acts  as  the  CMC’s  agent  in  this  process.  Part  of  MCCDC’s  overall  mission  is  to 
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translate  deficiencies  and  desired  capabilities  into  operational  requirements.  Meanwhile, 
the  mission  of  MARCORSYSCOM,  simply  stated,  is  to  take  a  validated  requirement  and 
turn  it  into  reality,  in  the  form  of  warfighting  weapon  systems  and  equipment.  The  CG, 
MCCDC  acts  as  the  Commandant’s  agent  in  developing  requirements  while  the 
Commander,  MARCORSYSCOM  acts  as  his  agent  in  acquiring  the  systems  that  fulfill 
those  requirements.  Clear  boundaries  between  requirements  determination  (CG, 
MCCDC)  and  acquisition  (COMMARCORSYSCOM)  exist  to  effectively  translate 
operational  needs  into  stable  and  affordable  acquisition  programs. 

The  Program  Managers  (PMs)  are  responsible  for  directing  the  efforts  of 
acquiring  the  systems  to  fulfill  the  validated  requirements.  They  are  responsible  for 
taking  the  requirement  from  concept  to  an  operational  system.  According  to  the  “USMC 
PM  Acquisition  Procedures  Handbook,”  in  broad  terms,  the  Program  Managers  have 
three  major  responsibilities:  “Cost,  Schedule,  and  Performance.”  It  should  be  noted  that 
the  handbook  mentions  “logistical  supportability”  as  a  part  of  performance  criteria  for 
which  program  managers  are  responsible  while  indicating  that  Integrated  Logistics 
Support  is  the  process  by  which  to  achieve  such  criteria  (USMC  PM  Acquisition 
Procedures  Handbook,  Chapter  1). 

With  the  inclusion  of  the  PM,  we  have  completed  the  streamlined  program 
decision  relationship  in  the  acquisition  hierarchy:  PM  to  CMDR,  MARCORSYSCOM, 
to  CAE  (NAE),  to  DAE.  Figure  3.1  generically  depicts  the  Marine  Corps  participants  in 
the  acquisition  process,  from  generation  of  the  requirement  and  program  initiation,  to 
fielding  and  post- deployment  support. 
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Figure  3.1.  Participants  in  the  Acquisition  Process.  (From:  USMC  PM  Acquisition 

Procedures  FInbk,  p.  1-14) 

2.  Acquisition  Phases  and  Milestones 

Along  with  the  recent  changes  to  the  DoD  Directive  5000  series,  a  new  DoD 
Systems  Acquisition  Process  model  was  created  which  was  intended  to  deliver  advance 
technology  to  the  warfighters  faster,  reduce  total  ownership  costs  and  improve 
affordability,  and  deploy  interoperable  and  supportable  systems.  Some  professionals  may 
argue  that  there  is  little  significant  difference  between  the  old  and  new  models  depicted  in 
Figure  3.2  aside  from  the  stages  and  milestones  renamed.  However,  others  point  out  that 
the  new  model  integrates  testing  and  evaluation  throughout  the  system;  allows  for 
“evolutionary  developments”  based  on  time-phased  (ORD)  requirements;  offers  multiple 
process  paths  or  entry  points  into  the  process  depending  on  conceptual  and  technical 
maturity  of  the  existing  system;  separates  technology  development  from  system 
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integration;  ensures  “entrance  criteria”  before  entering  the  next  phase  which  serves  as  a 
gate  for  the  Milestone  Decision  Authority  to  decide  if  the  program  should  continue; 
includes  operation,  support,  and  disposal  as  par  t  of  the  acquisition  process;  and  requires 
full  funding  at  system  development  vice  program  definition,  creating  more  competition 
between  competitors.  Despite  which  model  a  program  is  guided  by,  DoD  controls  the 
acquisition  process  through  a  series  of  tailorable  Milestones  and  Phases  that  serve  as 
decision  points  and  goals  to  be  achieved.  Additionally,  phases  help  focus  the  effort  and 
define  the  necessary  activities  for  effective  management.  However,  due  to  the  dynamic 
nature  of  DoD  acquisitions,  Program  Management  must  remain  flexible  (NPS  MN3331 
Class  Notes,  “Principles  of  Systems  Acquisition  and  Program  Management”). 
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Figure  3.2.  The  System  Acquisition  Models. 


Throughout  the  remainder  of  this  thesis,  future  references  of  the  phases  and 
milestones  most  often  cite  the  previous  model  due  to  the  fact  that  the  systems  examined 
in  this  study  were  procured  under  such  processes. 

a.  Acquisition  Program  Baseline  {APB) 

Each  program  has  an  APB  that  defines  the  cost,  schedule,  perf  ormance, 
and  supportability  measures  that  it  must  meet,  with  thresholds  and  objectives  defined  that 
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serve  as  boundary  parameters  within  which  the  PMs  operate.  The  APB  serves  as  a 
“contract”  of  sorts  between  the  PM  and  the  MDA.  Reliability  related  paramet  ers  such  as 
MTBF,  A0,  and  MTBM  exist  for  each  program  either  in  the  Performance  or 
Supportability  sections  of  the  APB.  The  acquisition  program  baseline  status  of  each 
program  is  reviewed  once  a  quarter  and  at  major  reviews  (Ryan,  p.  36). 

b.  Acquisition  Decision  Memorandum  (ADM) 

When  a  program  reaches  a  major  milestone  or  experiences  a  significant 
change  in  its  program  parameters,  the  outcome  is  documented  in  an  ADM.  The  ADM 
serves  to  document  decisions  made  by  the  MDA,  and  typically  includes  additional 
directive  statements  that  the  PM  must  comply  with.  The  Acquisition  Decision 
Memorandum  statements  and  directives  are  an  opportunity  for  the  MDA  to  encourage  the 
achievement  or  improvement  of  reliability  levels,  while  placing  exit  criteria,  constraints, 
or  follow-on  actions  related  to  reliability  on  the  programs. 

3.  Requirements  Generation  Process 

As  this  section  will  indicate,  the  Requirements  Generation  Process  can  serve  as  an 
initial  primary  tool  for  the  Marine  Corps  to  document  quantifiable  system  reliability 
requirements  in  the  Operational  Requirements  Document  (ORD)  in  the  form  of  Key 
Performance  Parameters  (KPP).  Reliability  requirements  definition  is  the  translation  of 
warfighters’  operational  requirements  into  specific  reliability  requirements  that  can  be 
defined,  designed  to,  and  measured.  The  requirements  definitions  are  incorporated  in 
written  specifications  that  contain  numerical  statements  of  required  reliability  and  precise 
description  of  the  performance  and  environmental  requirements  that  must  be  met  to 
achieve  the  numerical  reliability  requirements.  Close  attention  must  be  given  to  such 
reliability  requirements  because  they  are  eventually  used  as  contractual  and  acquisition 
devices  to  assure  mission  success  and  performance  over  time  (Reliability, 
Maintainability,  and  Supportability  Guidebook,  SAE,  p.  73). 

a.  Mission  Needs  Statement  (MNS) 

All  acquisition  programs  are  based  on  identified,  documented,  and 
validated  mission  needs,  resulting  from  ongoing  assessments  of  current  and  projected 
capability  with  respect  to  changing  military  threats  and  the  National  Security  Strategy 
(NSS).  Within  the  Marine  Corps,  part  of  MCCDC’s  overall  mission  is  to  translate 
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deficiencies  and  desired  capabilities  into  operational  requirements.  Requirements 
determination  and  revision  follow  an  established  process,  beginning  with  the  Capability 
Review  System  within  MCCDC  where  deficiencies  are  manifested  by  the  Fleet  Marine 
Force  (FMF)  through  Fleet  Operational  Need  Statements  (FONS),  the  Marine  Corps 
Lessons  Learned  System  (MCLLS),  Mission  Area  Analysis  (MAA),  and  the  Marine 
Corps  Master  Plan  (MCMP).  Additionally,  the  natural  expiration  of  service  life  of 
equipment  is  factored  into  the  process.  A  material  solution  to  a  deficiency  begins  with  a 
broad  statement  of  the  requirement  as  outlined  in  a  Mission  Need  Statement  (MNS), 
developed  by  MCCDC,  and  sent  to  the  Assistant  Commandant  of  the  Marine  Corps 
ACMC  for  approval.  The  MNS  is  a  non -system  specific  statement  of  operational 
capability  need  written  in  broad  operational  terms.  It  is  non-specific  by  design  and  offers 
a  materiel  solution  in  one  of  three  ways:  improvements  to  an  existing  system, 
procurement  of  a  non- developmental  item,  or  begin  a  new  research  and  development 
program.  Subsequent  approval  and  signature  of  the  MNS  by  the  ACMC  constitutes  a 
“validated  requirement”  and  initiates  Milestone  A.  Following  the  Mission  Need 
Statement,  MCCDC  performs  individual  Analysis  of  Alternatives  (AoA),  and  although 
not  a  requirements  document,  it  forms  the  basis  for  an  Operational  Requirements 
Document  (ORD),  which  is  also  drafted  by  MCCDC  (USMC  PM  Acquisition  Procedures 
Handbook,  p.  1-6).  It  is  through  the  AoA  that  an  approach  is  formulated  to  set  and  refine 
life  cycle  cost  objectives. 

b.  Operational  Requirements  Document 

The  ORD  is  a  key  document  in  the  acquisition  process,  for  it  translates  the 
MNS  into  more  detailed  and  refined  performance  capabilities  and  characteristics  of  a 
proposed  concept  or  system.  To  do  so,  the  ORD  defines  the  requirement,  states  the 
numbers  of  systems  and  where  they  should  be  fielded,  and  describes  the  specific 
operational  capabilities  required.  MCCDC  acts  as  the  Combat  Developer,  and  develops 
the  Operational  Requirements  Document,  which  details  the  required  system  capabilities 
and  characteristics  to  include  the  user’s  definition  of  system  reliability  parameters  in 
operational  terms.  MCCDC  is  ultimately  responsible  for  defining  the  requirements 
relative  to  the  reliability  of  the  system.  It  is  at  this  stage  that  defining  the  “essential 
qualitative  and  quantitative  readiness  and  logistics  supportability  requirements  in 
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operational  concepts  and  requirements  documents  is  the  most  effective  way  for  users  to 
influence  the  design  of  their  systems”  (Department  of  Air  Force,  Instruction  10-602, 
1994).  Typically,  this  is  defined  in  terms  of  operational  availability  and  mission  duration 
needs.  As  directed  in  DoD  5000.2-R,  these  operational  performance  parameters  are  to  be 
stated  as  Objectives  and  Thresholds.  Section  2.6  of  DoD  5000.2-R  states 

supportability  factors  are  integral  elements  of  program  performance 
specifications.  However,  support  requirements  are  not  to  be  stated  as 
distinct  logistics  elements,  but  instead  as  performance  requirements  that 
relate  to  a  system’s  operational  effectiveness,  operational  suitability,  and 
life-cycle  cost  (DoD  5000.2-R). 

Reliability,  along  with  cost,  schedule,  and  performance,  should  act  as 
equal  partners  in  the  requirements  generation  process.  An  effective  way  to  ensure  that  a 
system  maximizes  its  operational  availability  is  to  include  robust  reliability  goals  in  the 
ORD. 

At  each  milestone,  beginning  with  program  initiation,  thresholds  and 
objectives  initially  expressed  as  measures  of  effectiveness  (MOEs)  and  minimum 
acceptable  requirements  for  the  proposed  concept  or  system  are  documented  by  the  user 
or  the  user’s  representative  in  the  ORD  to  quantify  system  level  performance.  Thresholds 
and  objectives  in  the  ORD  consider  the  results  of  the  analysis  of  alternatives  and  the 
impact  of  affordability  constraints  (DSMC,  “Acquisition  Logistics  Guide”,  p.  5-2).  The 
Combat  Developer’s  definition  of  the  intended  reliability  requirement  is  an  essential 
element  in  establishing  the  basis  for  any  successful  reliability  program.  Whether  the 
requirements  result  from  the  needs  of  the  user  or  from  internal  goals  identified  by  a 
design  or  project  organization,  well-defined  requirements  are  needed.  Conversely,  poorly 
defined  requirements  lead  to  conflicts  in  direction  and  inefficiencies  in  the  application  of 
engineering  and  management  resources.  If  the  requirements  are  defined  properly,  close 
adherence  to  the  ORD  is  necessary  for  a  successful  logistics  program  (Reliability, 
Maintainability,  and  Supportability  Guidebook,  SAE,  p.  42). 

Reliability  requirements  determination  is  not  accomplished  in  a  vacuum. 
In  fact,  developing  quantitative  operational  reliability  requirements,  like  all  other  ORD 
requirements,  is  a  collaborative  process  between  the  combat  developer  (MCCDC)  and  the 
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materiel  developer  (MARCORSYSCOM)  using  Integrated  Product  Teams  (IPTs).  This 
process  provides  a  balanced  solution  between  the  best  estimate  of  what  is  required  to 
meet  the  warfighter’s  effectiveness,  suitability,  and  survivability  needs,  and  that  which  is 
actually  affordable  and  technically  achievable  within  program  funding,  risk,  and  time 
constraints  (Ryan,  p.  13). 

c.  Key  Performance  Parameters 

While  the  ORD  serves  to  establish  minimum  acceptable  operational  values 
for  broad  performance  parameters,  the  Marine  Corps  has  the  opportunity  to  include 
quantifiable  and  understandable  reliability  requirements  as  Key  Performance  Parameters 
(KPPs)  in  the  ORD.  A  KPP  is  a  capability  or  characteristic  that  is  so  significant  that 
failure  to  meet  the  threshold  can  be  cause  for  the  concept  or  system  selection  to  be 
reevaluated  or  the  program  to  be  reassessed  or  terminated.  By  placing  reliability 
requirements  as  KPPs  in  the  ORD,  contractors  would  be  required  to  test  to  reliability. 
Such  KPPs  would  likely  ensure  adequate  logistics  weight  in  source  selection. 
Unfortunately,  reliability  (as  well  as  availability  and  maintainability)  requirements  are 
usually  not  KPPs,  and  when  there  are  cost  or  schedule  overruns,  reliability  is  sacrificed. 
In  reality,  reliability  KPPs  should  be  expressed  with  both  threshold  (minimally  accep  ted 
values)  and  objectives  (what  the  user  desires  and  what  the  PM  is  attempting  to  obtain). 
Then,  given  a  system’s  reliability  goal  that  is  clearly  defined  by  the  Combat  Developer  as 
a  KPP  in  the  ORD,  the  designer  understands  what  reliability  the  system  should  be 
“designed  to.” 

Part  of  the  intent  of  new  5000  series  and  the  new  acquisition  model  is  to 
reduce  Total  Ownership  Costs  (TOC)  by  minimizing  the  number  of  mission -oriented  Key 
Performance  Parameters.  Upper  levels  of  DoD  believe  that  this  maximizes  the  PM’s  and 
contractor’s  flexibility  to  make  cost/performance  tradeoffs  without  the  unnecessary 
higher-level  permission,  proving  to  be  essential  to  achieving  cost  objectives.  Therefore, 
the  number  of  threshold  items  in  requirements  documents  and  acquisition  program 
baselines  are  strictly  limited.  The  threshold  values  represent  true  minimums,  and  the 
requirements  should  be  stated  in  terms  of  capabilities  rather  than  technical  solutions  and 
specifications.  While  reliability  related  KPPs  typically  are  not  in  the  ORD,  many 
professionals  will  argue  that  they  should  be  a  mandatory  part  of  the  ORD. 


50 


4.  Contracting 

As  the  previous  section  indicates,  to  attain  a  desired  combat  capability,  or 
operational  thresholds  and  goals,  requirements  must  be  communicated  in  the  ORD  in 
clear  operational  terms,  a  responsibility  of  the  Combat  Developer.  The  reliability 
objectives  must  then  be  translated  into  quantifiable  and  verifiable  contractual  terms 
traceable  back  to  the  operational  requirements.  The  Materiel  Developer  must  adequately 
translate  the  system  operational  terms  into  viable  contractual  terms  understood  by  all 
parties  involved  to  include  the  user,  the  program  office,  and  the  contractor  so  that 
compliance  can  be  adequately  monitored  and  enforced.  Previously  in  the  traditional 
acquisition  process,  the  Materiel  Developer  could  insert  reliability  requirements  in  the 
system  specification  and  development  specifications  and  then  incorporate  tasks  in  the 
statement  of  work  (SOW),  allowing  the  contractor  to  conduct  a  disciplined  reliability 
program  to  achieve  the  requirements  (SD-2  “Buying  Commercial  and  .  .  .”,  Ch.  6). 
However,  recent  policy  changes  resulting  from  the  military  specifications  and  standards 
reform  in  1994  has  led  to  the  incorporation  of  a  performance -based  approach  to  reliability 
in  Request  for  Proposals,  eliminating  the  use  of  “how  to”  reliability  standardization 
documents. 

a.  Performance  Specifications 

The  MNS,  AoA,  and  ORD  are  provided  to  the  Materiel  Developer 
(MARCORSYSCOM)  for  performance  specification  development,  or  the  translation  of 
user  requirements  into  performance  specifications  that  should  be  understandable  to 
potential  contractors.  Performance  specifications  eventually  become  major  pieces  to  the 
Request  for  Proposal  (RFP)  and  the  contract,  and  thus,  they  are  to  clearly  state  what  the 
system  must  do,  how  well  it  must  perform,  under  what  circumstances  and  conditions,  and 
identify  other  constraints.  However,  performance  specifications  do  not  dictate  to 
contractors  how  to  achieve  the  required  performance. 

It  is  important  to  note  that  developmental  testing  is  conducted  to 
contractual  and  performance  specifications,  while  operational  testing  is  conducted  to 
ORD  operational  thresholds.  “The  operational  user,  the  program  offices,  and  the 
contractor  often  get  very  confused  over  the  process  of  translating  ORD  (operational 
threshold)  numbers  to  contract  (performance)  specifications  and  vice  versa”  (DSMC, 
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“Acquisition  Logistics  Guide”,  p.  10-6).  The  user  or  warfighter  often  has  various 
measures  highlighted  in  the  ORD  that  must  be  translated  by  the  program  office  into 
performance  specifications.  Table  3.1  provides  a  sample  of  user  measurements  compared 
to  the  common  contractual  reliability  specification  of  MTBF. 


USER  OBJECTIVE  AREA 

RELIABILITY  (MTBF) 

- Operational  Effectiveness - 

Increase  Readiness 

Mean  Time  Between  Downing 

Events  (MTBDE) 

Increase  Mission  Success 

Mean  Time  Between  Critical 

Failures  (MTBCF) 

- Ownership  Costs . 

Decrease  Maintenance  Personnel  Costs  Mean  Time  Between 

Maintenance  Actions  (MTBM) 

Decrease  Logistic  Support  Costs 

Mean  Time  Between  Removals 
(MTBR) 

Table  3.1.  Measures  of  Systems  Readiness.  (From:  DSMC,  “Acquisition  Logistics 

Guide”,  p.  10-6) 

There  must  be  a  clear  connection  between  the  defined  operational 
reliability  requirements  in  the  ORD,  created  by  the  Combat  Developer  and  the 
performance  specifications  completed  by  the  Materiel  Developer  in  the  terms  of  the 
contract.  Conversion  of  commonly  used  operational  terms  such  as  MTBM  and  MTBCF 
must  be  made  to  enable  translation  to  parameters  that  can  be  specified  in  contracts  as  well 
as  verified  in  testing.  In  doing  so,  one  of  the  major  difficulties  is  attempting  to  merge 
contractual  reliability  and  operational  reliability. 
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CONTRACTUAL  RELIABILITY 


OPERATIONAL  RELIABILITY 


•  Used  to  define,  measure  and  evaluate 
contractor’s  program 

•  Derived  from  operational  needs 

•  Selected  such  that  achieving  them  allows 
projected  satisfaction  of  operational 
reliability 

•  Expressed  in  inherent  values 

•  Accounts  only  for  failure  events  subject  to 
contractor  control 

•  Includes  only  the  design  and 
manufacturing  characteristics 

TYPICAL  TERMS: 

•  MTBF  (Mean  Time  Between  Failure) 

•  Mission  MTBF  (sometimes  called 
MTBCF) 


•  Used  to  describe  reliability  performance 
when  operated  in  the  planned  environment 

•  Not  used  for  contract  reliability 
requirements  (requires  translation) 

•  Used  to  describe  the  required  level  of 
reliability  performance 

•  Includes  the  combined  effects  of  item 
design,  quality,  installation/repair 
environment,  maintenance  policy,  repair,  etc. 

TYPICAL  TERMS: 

•  MTBM  (Mean  Time  Between 
Maintenance) 

•  MTBD  (Mean  Time  Between  Demand) 

•  MTBR  (Mean  Time  Between  Removal) 

•  MTBCF  (Mean  Time  Between  Critical 
Failure) 


Table  3.2.  Contractual  vs.  Operational  Reliability.  (From:  Reliability  Engineers 

Toolkit:  Rome  Laboratory) 

b.  Source  Selection  Factors 

The  Marine  Corps  also  has  the  opportunity  to  use  reliability  as  a  factor  in 
source  selection,  arguably  the  most  important  contractor  motivational  factor.  In  source 
selection  for  a  modified  or  new  system,  reliability  must  be  singled  out  as  a  specific 
evaluation  sub  factor.  Reliability  should  be  a  performance  requirement  used  in  the 
solicitation  process.  In  other  words,  reliability  plans  and  goals  should  always  be  a  source 
selection  evaluation  sub  factor. 

In  the  solicitation  process.  Request  For  Proposals  (RFPs)  include  a  strict 
minimum  number  of  critical  performance  criteria  that  force  contractors  to  meet  the 
desired  program  objectives.  The  desired  reliability  and  cost  objectives  can  be  used  as  a 
management  or  leveraging  tool  that  forces  contractors  to  meet  such  objectives.  Because 
potential  suppliers  are  competing  for  a  contract,  there  is  a  natural  tendency  for  contractors 
to  emphasize  their  strengths  while  concealing  their  weaknesses.  While  it  is  often  useful 
to  utilize  contractor  testing  results,  it  is  important  to  ascertain  their  capabilities  through 
probing,  questioning,  and  eventually,  independent  military  testing  as  will  be  discussed  in 
an  upcoming  section. 
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c.  Contracts,  Clauses,  Warranties,  and  Incentives 
After  reliability  requirements  have  been  established,  “the  apportioned 
values  (MTBF,  MTTR,  and/or  relevant  criteria)  should  be  included  in  appropriate 
sections  of  procurement  specifications,  critical  item  specifications,  and  contractor  end- 
item  specifications”  (DSMC,  “Designing  Quality  Into  Defense  Systems”,  p.  17).  The 
contractor  and  designer  must  clearly  understand  every  critical  requirement  the  system 
must  meet  so  that  if  needed,  trade-offs  can  be  executed  based  on  government  priorities. 

While  predicted  reliability  typically  comes  from  contractor  claims,  the 
DoD  needs  some  confidence  level  that  it  is  a  good  system  of  merit  for  predicted 
reliability.  The  Materiel  Developer  must  attempt  to  contract  to  a  given  or  specified 
reliability  confidence  level  or  to  a  commitment  to  a  specified  target  operational 
availability  in  an  effort  to  hold  contractors  accountable  to  their  original  reliability 
estimates.  When  dealing  with  contractors’  predicted  reliability,  the  null  hypothesis  that 
the  estimate  is  incorrect  should  be  ass  umed  until  proven  otherwise. 

Additionally,  the  contracts  resulting  form  the  source  selection  should  have 
incentive  clauses  related  to  the  level  of  reliability  achieved  and  verified.  Warranties  can 
be  utilized  to  hold  contractors  responsible  for  sustaining  in  the  operational  environment, 
the  performance  levels  which  have  been  contractually  agreed  to.  Then,  if  the  contractor 
does  not  meet  the  contractual  reliability  goals,  reliability  shortfalls  should  be  considered  a 
latent  defect.  Additionally,  incentives  such  as  cash  rewards  can  be  used  to  motivate 
contractors  to  exceed  minimum  program  requirements  and  predetermined  thresholds  for 
reliability.  However,  the  use  of  contract  warranties  and  incentives  sometimes  imposes 
unrealistic  data  collection  demands  on  the  operational  user  and  field  maintenance 
organization,  making  it  difficult  to  enforce  the  warranty  provisions.  The  operational 
scenario  must  be  evaluated  to  determine  if  warranty  conditions  are  practical. 
Unfortunately,  in  the  past, 

PMs  often  disregard(ed)  logistics  contract  considerations,  such  as 
identifying  logistics  deliverables  and  creating  the  logistics  input  to  the 
Statement  of  Work  (SOW),  as  long-term  issues  that  are  less  important  than 
the  immediate  problems.  As  a  result,  logistics  concer  ns  are  (were)  often 
deferred  for  later  resolution  (DSMC,  “Acquisition  Logistics  Guide”,  p.  17  - 
8). 
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One  of  the  more  recent  trends  has  been  experimentation  with  Contractor 
Logistics  Support  (CLS),  which  has  shown  indicators  of  lower  costs  and/or  increased 
readiness.  Under  CLS,  the  performance  of  maintenance  and/or  materiel  management 
functions  for  DoD  systems  is  conducted  by  commercial  activities.  A  discussion  of  the 
benefits  and  challenges  of  CLS  are  beyond  the  scope  of  this  thesis. 

Another  recent  initiative  has  been  the  use  of  Performance  Based  Logistics 
(PBL)  and  Performance  Based  Payments  (PBP).  This  strategy  is  a  method  of  providing 
financing  to  contractors,  performing  under  fixed -priced  contracts,  where  performance 
based  payments  are  given  upon  the  achievement  of  specific  events  or  accomplishments 
that  are  defined  and  valued  in  advance  by  the  parties  to  the  contract,  rather  than  being 
tied  to  and  based  upon  incurred  costs  of  performance  (DoD  Users  Guide  to  Performance 
Based  Payments,  Chap  1).  It  is  an  integrated  acquisition  and  logistics  process  for  buying 
weapon  system  capability  and  instead  of  buying  set  levels  of  spares,  repairs,  tools,  and 
data,  there  is  a  focus  on  buying  a  predetermined  level  of  availability  to  meet  the 
warfighters’  objectives.  In  PBL,  the  contract  requirement  is  specified  in  service  terms. 
For  example,  the  number  of  hours  at  a  given  cost  per  hour  and  customer  response-type 
metrics  such  as  availability  may  be  used  to  describe  the  service.  When  properly 
incetivized,  the  PBL  provider  strives  for  continuous  improvement  in  reliability  to 
eliminate  his  maintenance  efforts  altogether. 

The  bottom  line  remains  that, 

the  well-meaning  but  ineffectual  philosophy  often  applied  to  reliability  - 
‘we  will  do  the  best  we  can’  should  be  replaced  by  a  contractual  obligation 
in  the  form  of  quantitative  system  reliability  requirements  that  forces 
contractors  to  consider  reliability  equally  with  other  system  parameters 
such  as  performance,  weight,  cost,  etc  (Kececioglu,  “Reliability 
Engineering  Handbook,”  Chap.  15). 

To  do  so,  contracts  and  contract  warranty  clauses  must  be  specific  while 
the  user,  the  program  office,  and  the  contractor  must  understand  and  agree  to  the 
reliability  terms  in  both  the  ORD  and  contract  specification.  Ultimately,  reliability  and 
logistics  program  success  are  a  direct  reflection  of  contract  success. 
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5.  Conceptualization,  Design,  and  Development:  Systems  Engineering 
Process 

System  effectiveness  and  cost  are  the  drivers  in  design  decision,  and  given  the 
trend  towards  the  development  of  increasingly  complex  weapon  systems,  it  is  obvious 
that  reliability  cannot  be  ignored  and  left  as  a  matter  of  chance  when  considering  design. 
Instead,  reliability  must  be  consciously  and  proactively  built  into  systems  through 
effective  design  and  manufacturing  practices.  The  method  for  doing  so  is  the  systems 
engineering  process  (SEP),  which  is  used  to  translate  operational  needs  and  requirements 
into  a  system  solution  that  includes  the  design,  manufacturing,  test  and  evaluation, 
support  processes,  and  products.  This  includes  transforming  operational  needs  and 
requirements  into  an  integrated  system  design  solution  through  concurrent  considerations 
of  all  like-cycle  needs. 

A  major  goal  and  function  of  the  systems  engineering  process  is  the  achievement 
of  a  proper  balance  cost,  schedule,  risk,  and  performance  (to  include  readiness  and 
supportability).  To  do  so,  supportability  analyses  are  conducted  as  an  integral  part  of  the 
systems  engineering  process,  beginning  at  program  initiation  and  continuing  throughout 
system  development.  Supportability  analyses  form  the  basis  for  related  design 
requirements  included  in  the  system  specification  and  for  subsequent  decisions 
concerning  how  to  support  the  system  in  the  most  cost-effective  manner  over  its  entire 
life  cycle  (DSMC,  “Acquisition  Logistics  Guide”,  pp.  3-10  -  3-12). 

The  system  engineering  process  is  an  iterative  interdisciplinary  problem  solving 
methodology  that  allows  the  Government  and  the  contractor  to  create  an  integrated  and 
life  cycle  balanced  set  of  system  product  and  process  solutions  based  on  Government 
performance  specifications  and  system  requirements.  The  process  serves  to  determine 
critical  interfaces  for  system  integration  by  progressively  decomposing  system 
requirements  into  performance  specifications  and  defining  all  subsystems,  assemblies, 
and  parts.  As  a  result,  the  SEP  assists  in  verifying  that  the  system  design  meets  user 
requirements.  While  the  system  engineering  process  is  typically  applied  at  the  prime 
contractor  level,  relevant  requirements  are  passed  down  to  the 
subcontractor/supplier/vendor  levels.  Figure  3.3  illustrates  the  iterative  nature  of  this 
process. 
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Figure  3.3.  Systems  Engineering  Process.  (From:  DSMC  Program  Managers  Tool  Kit, 

p.  65) 


Application  of  the  system  engineering  process  to  reliability  design  is 
accomplished  through  a  structured  process  of  functional  analysis,  design  synthesis, 
alternative  exploration,  trade-off  evaluation,  and  decision  making  which  is  iterated 
throughout  the  design  process  to  achieve  the  desired  levels  of  performance.  Maximum 
benefit  accrues  through  the  integration  of  reliability  into  the  system  engineering  process 
during  early  development  activities  since  most  of  the  system  life  cycle  costs  are 
determined  in  the  early  phases  of  development.  The  SEP  is  based  on  the  Integrated 
Product  Process  Development  (IPPD)  framework,  which  is  a  management  technique  that 
simultaneously  integrates  all  essential  acquisition  activities  through  the  use  of  multi¬ 
disciplinary  teams  to  optimize  the  design,  manufacturing,  and  supportability  processes. 
The  multi-disciplinary  aspect  of  SEP  serves  as  an  effective  way  to  get  the  various 
disciplines  working  together.  Thus,  systems  engineering  programs  are  required  by  Do  D 
5000. 2-R  for  all  Acquisition  Category  (ACAT)  programs. 

The  balanced  integration  of  logistics  considerations  into  the  systems  engineering 
process  is  imperative  from  the  onset.  System  reliability,  maintainability,  and 
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supportability  must  be  key  elements  of  the  tradeoff  and  design  criteria  in  each  stage  of 
the  process  as  design  considerations  will  inevitably  be  in  conflict  with  reliability, 
maintainability,  and  supportability  goals.  When  such  conflicts  do  occur,  the  latter  goals 
must  be  considered  equally  with  acquisition  cost,  schedule,  and  performance.  The 
logistician  must  be  a  principal  player  in  the  development  process  as  indicated  in  the 
below  excerpt  from  MIL  HDBK-502. 

Unfortunately,  acquisition  logistics  (supportability)  objectives  often 
conflict  with  other  design  objectives  like  speed,  range,  size,  etc.  How  is 
this  inevitable  conflict  resolved?  Early  in  the  process,  the  issue  of 
tradeoffs  must  be  raised  during  the  analysis  of  proposed  concepts.  Careful 
use  of  tradeoff  studies  will  guide  the  engineers  and  the  logisticians  in 
finding  the  optimal  design  —  one  which  balances  design  objectives  with 
supportability  requirements.  Tradeoffs  are  an  essential  part  of  the  design 
process. 

The  result  of  this  early  collaboration  between  engineering  and  logistics 
personnel  is  a  specification  that  prescribes  performance  requirements  to  be 
achieved. 

The  challenge  is  to  ensure  that  supportability  is  integrated  into  the 
program  from  the  beginning  phases.  The  early  design  phases  of  a  project, 
when  things  change  rapidly,  may  seem  of  little  interest  to  logisticians,  and 
their  attendance  at  engineering  design  reviews  may  seem  a  waste  of  time. 
Actually  this  period  has  far  reaching  logistics  impact.  During  this  phase 
the  logisticians  can  use  the  leverage  of  early  program  involvement  to 
identify  approaches  that  will  significantly  lower  life  cycle  costs.  They  may 
be  able  to  catch  an  exorbitantly  expensive  material  or  time-consuming 
maintenance  process  before  it  has  become  integrated  into  the  system  (MIL 
HDBK-502,  6.2.1). 

6.  Test,  Production,  and  Verification 

One  must  learn  by  doing  a  thing;  for  though  you  think  you  know  it,  you 
have  not  certainty  until  you  try.  -  Sophocles 

Once  a  system  has  been  selected,  it  is  imperative  to  demonstrate,  through  testing, 
that  system  capabilities  meet  contract  specifications  and  satisfy  mission  needs. 
Specifically,  as  the  proceeding  sections  will  indicate,  reliability  demonstrations  and 
consequential  logistics  and  supportability  factors  must  be  included  as  part  of  the  testing, 
production,  and  verification  of  new  weapon  systems.  Unfortunately,  demonstration  of 
required  reliability  performance  levels  prior  to  system  fielding  is  often  a  challenge. 
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Because  the  logistical  support  system  will  be  built  upon  the  accepted  reliability 
estimates,  the  verification  of  reliability  figures  is  crucial.  It  is  during  testing  that  DoD 
organizations  must  validate  the  contactors’  reliability  estimates  in  an  effort  to  avoid 
future  unexpected  life  cycle  cost,  supportability,  and  readiness  problems  as  weapon 
systems  mature.  Based  on  system  design  and  its  reliability  and  maintainability 
predictions,  the  PM  office  will  determine  the  number  of  spares  of  each  particular  type 
that  will  be  purchased,  what  support  equipment  will  be  used,  whether  new  equipment  will 
be  procured,  the  types  of  skills  needed  and  the  varying  skill  levels  required  as  well  as 
other  manpower  considerations,  funding  requirements,  and  POM  considerations.  If  the 
USMC  is  basing  its  Integrated  Logistical  Support  Packages  (ILSP)  upon  initial  contrac  tor 
reliability  estimates  prior  to  fielding,  it  is  imperative  to  have  accurate  reliability 
estimates.  Unfortunately,  contractor  reliability  estimates  (of  systems  and  their 
components)  are  sometimes  far  different  from  the  actual  achieved  reliability  of  fie  lded 
systems,  causing  possible  catastrophic  effects,  readiness  degradation,  or  enormous  and 
unexpected  Life  Cycle  Costs  which  eventually  create  additional  need  for  O&S  dollars  in 
later  years. 

Testing  (to  include  reliability  testing)  serves  several  general  purposes:  1.)  to 
gauge  the  progress  being  made  when  a  concept  is  being  translated  into  an  actual  product; 
2.)  to  mature  the  system  by  revealing  design  and  process  deficiencies  so  that  corrective 
action  may  occur  when  it  is  least  costly  to  fix;  and  3.)  to  determine  compliance  with  the 
requirement  and  determine  operation  suitability  through  formal  qualification  or 
demonstration  testing  prior  to  fielding.  There  are  many  types  and  levels  of  technical  and 
operational  tests  that  are  available  and  used  by  both  contractors  and  the  government. 
While  discussion  of  such  tests  are  beyond  the  scope  of  this  thesis,  some  of  the  common 
tests  include  but  are  not  limited  to:  simulations,  environmental  stress  testing,  accelerated 
life  testing,  reliability  development/growth  testing  (RD/GT),  reliability  qualification 
(RQT)/demonstration  testing  (RDT),  developmental  test  and  evaluation  (DT&E), 
operational  test  and  evaluation  (OT&E),  early  user  test  (EUT)/Limited  User  Test  (LUT), 
initial  operational  test  (IOT),  life  fire  test  and  evaluation  (LFT&E),  follow-on  test  (FOT), 
and  many  more.  For  general  background  purposes,  the  next  sections  will  briefly  examine 
DT&E  and  OT&E,  the  two  most  general  categories  that  of  DoD  testing.  Table  3.3  serves 
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to  further  distinguish  between  developmental  and  operational  testing  while 
complementing  the  proceeding  sections. 


DEVELOPMENTAL  TEST  & 
EVALUATION 

•  Technical  performance 
measurement 

•  Developing  agency  responsibility 
(PM) 

•  Technical  personnel 

•  Limited  test  articles  /  each  test 

•  Controlled  environment 

•  All  types  of  test  articles  / 
prototypes 

•  Government  /  contractor 
involvement 


OPERATIONAL  TEST  & 
EVALUATION 

•  Designed  to  obtain  operational 
effectiveness  /  suitability  data 

•  Operational  Test  Agency 
Responsibility  (MCOTEA  for 
USMC) 

•  "Typical”  user  personnel 

•  Realistic  combat  environment  and 
threats 

•  "Production  Representative”  test 
articles  /  LRIP  items 

•  Contractor  involvement  restricted 


Table  3.3.  DT&E  and  OT&E  Comparisons.  (From:  DSMC  PM  Toolkit,  p.  51) 
a.  Developmental  Test  and  Evaluation 

The  overall  goal  of  developmental  testing  is  to  determine  whether  the 
weapon  system  meets  the  technical  contract  and  performance  specifications.  DT&E  is  a 
method  for  the  PM  to  make  the  system  work,  to  verify  contractor  claims  and  predictions, 
and  to  influence  the  system  design.  Such  testing  assists  in  the  development  and 
maturation  of  products,  product  elements,  and  support  processes  and  is  utilized  to  verify 
the  status  of  technical  progress,  verify  that  design  risks  are  minimized,  and  certify 
readiness  for  initial  operation  testing.  While  both  contractors  and  Government  personnel 
are  involved  in  DT&E,  the  tests  are  generally  accomplished  by  engineers,  technicians,  or 
operator-maintainer  test  personnel  in  a  controlled  environment  to  facilitate  failure 
analysis. 

The  feedback  provided  by  developmental  testing  allows  those  personnel 
involved  in  the  systems  engineering  process  to  analyze  the  test  results  and  implement 
required  adjustments  before  testing  again.  As  expected,  reliability  engineers  and 
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logisticians  play  a  critical  role  during  DT&E  through  the  IPT  process.  However,  the 
Program  Manager  ultimately  controls  the  DT  environment  and  is  provided  with  the  data 
throughout  the  testing  cycles,  enabling  the  PM  to  make  informed  managerial  decisions 
that  affect  the  reliability  of  the  final  product.  Developmental  testing  identifies 
capabilities  and  limitations  of  alternatives  and  comparisons  of  candidates.  The  PM 
typically  is  forced  to  make  cost-performance  trade-off  decisions  before  eventually 
certifying  that  the  system  is  ready  for  operational  test  and  evaluation  (OT&E). 

b.  Operational  Test  and  Evaluation 

Operational  testing  is  the  field  test  for  any  system  or  key  component  of  the 
weapon  system,  conducted  under  realistic  conditions,  to  determine  the  operational 
effectiveness  and  suitability  of  the  system  for  use  in  realistic  combat  conditions  by 
typical  military  users.  Operational  testing  should  determine  whether  minimum 
acceptable  operational  performance  requirements  (ORD  thresholds)  have  been  satisfied. 
Unlike  developmental  testing,  operational  testing  is  conducted  by  independent  military 
test  organizations  not  beholden  to  the  program  office,  which  represent  the  customers  or 
combat  units  that  will  ultimately  use  the  systems.  As  a  result,  operational  testers 
typically  have  more  independence  than  developmental  testers  as  they  provide  their  results 
to  Congress  as  well  as  to  senior  officials  in  the  services  and  the  Office  of  the  Secretary  of 
Defense  (GAO,  “A  More  Constructive  Test .  .  ”,  p.  1 1) 

c.  Testing  Summary 

Despite  what  category  or  level  of  testing  is  being  conducted,  credible  and 
properly  designed  tests  must  be  addressed,  conducted,  and  properly  evaluated  early  in  the 
development  process  for  results  to  be  useful.  However,  weapon  system  programs  have 
traditionally  suffered  from  persistent  problems  associated  with  late  or  incomplete  testing. 
While  discovery  of  problems  in  any  complex  product  (through  testing)  is  a  normal  and 
desired  part  of  the  developmental  process,  surprises  in  testing  or  repeated  occurrences 
often  polarize  organizations  into  proponents  and  critics  of  programs.  It  is  difficult  for 
weapon  system  programs  to  compete  for  approval  unless  the  system  offers  significantly 
better  performance  over  other  systems  while  remaining  within  available  funding  and 
scheduling  constraints.  As  a  result,  there  are  greater  incentives  for  PMs  to  “accept 
immature  technologies  and  make  optimistic  assessments  about  what  can  be  accomplished 
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with  limited  resources.”  Test  results  tend  to  become  scorecards  that  demonstrate  whether 
the  program  is  ready  to  proceed  or  to  receive  the  next  increment  of  funding.  In  the  DoD, 
unlike  in  the  commercial  sector,  testing  and  evaluation  is  more  for  the  benefit  of  the 
testers  and  decisionmakers  above  the  program  manager.  Thus,  managers  often  have 
incentives  to  postpone  difficult  tests  and  to  limit  open  communication  about  the  test 
results  (GAO,  “A  More  Constructive  Test .  .  ”,  p.  8-9). 

7.  Maintaining  Reliability  of  Fielded  Systems 

Managing  reliability  does  not  end  with  OT&E  and  fielding  of  the  system,  and 
instead,  reliability  must  be  continually  monitored  and  assessed  for  potential 
improvements  and  efficiencies  in  support  of  meeting  Marine  Corps  life  cycle  cost  and 
readiness  objectives.  In  fact,  once  a  system  is  fielded,  reliability  assessment  should 
become  a  permanent  part  of  sustainment  activities  conducted  by  Program  Management 
Offices  as  well  as  other  Life  Cycle  Management  organizations.  To  be  successful, 
reliability  growth  must  continue  during  the  customer-use  phase  by  coordinating  feedback 
from  the  warfighters  to  the  suppliers  and  by  supporting  necessary  corrective  actions.  Part 
of  Phase  III  (Production,  Fielding/Deployment,  &  Operational  Support)  responsibilities 
include  ensuring  fielded  systems  continue  to  meet  mission  requirements  throughout  their 
planned  life  cycles.  Specifically,  critical  systems  and  components  should  be  identified 
where  low  reliability  rates  are  degrading  readiness  and  causing  unnecessary  support 
costs. 

The  basic  policy  of  DOD  is  to  hold  contractors  responsible  for  quality  of  the 
products  through  quality  assurance  programs.  Quality  assurance  is  defined  in  DODD 
4155.1  as  “a  planned  and  systematic  pattern  of  actions  necessary  to  provide  adequate 
confidence  that  material,  data,  supplies,  and  services  conform  to  established  technical 
requirements  and  achieve  satisfactory  performance”  (DSMC,  “Designing  Quality  Into 
Defense  Systems”,  p.  8).  This  obviously  requires  a  plan  and  action,  which  must  be  based 
on  the  quality  requirements  as  outlined  in  the  ORD.  To  do  so,  it  is  recomm  ended  that  a 
program  use  the  reliability  requirements  stated  in  operational  requirements,  or  those 
resulting  from  trade-off  analysis,  as  a  baseline  for  reliability  assessment  to  be  compared 
with  actual  achieved  field  reliability.  However,  the  difficulty  remains  in  collecting, 
interpreting,  comparing  operational  (achieved)  reliability  with  contractual  reliability 
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measurements  as  illustrated  in  the  previous  Table  3.2.  Aside  from  the  essential  collection 
of  achieved  field  data,  original  contractor  estimates  and  requirements  must  be  retained  for 
comparison.  It  may  not  be  surprising  to  find  that  such  documentation  is  not  typically 
retained  and  is  difficult  to  locate. 

An  example  of  the  difference  between  inherent  (or  potential)  reliability  and 
achieved  value  is  shown  graphically  in  Figure  3.4.  The  operation  and  maintenance  of 
equipment  in  the  field  can  induce  these  effects  by  stressing  systems  beyond  predicted 
levels.  Additionally,  the  true  achieved  reliability  can  be  obscured  by  scheduled  and 
unscheduled  maintenance  actions  and  the  corresponding  incorrect  administrative  actions. 
Operational  contributors  to  such  overstresses  include  neglect,  unfamiliarity,  carelessness, 
and  mission  constraints.  Maintenance  actions  can  also  induce  defects  in  otherwise 
satisfactory  assemblies;  foreign  objects  introduced,  fasteners  improperly  engaged, 
contaminants  introduced,  improper  part  replacement,  improper  lubricants,  etc.  While  a 
major  effort  is  made  in  operations  to  reduce  the  effects  of  reliability  degradation  caused 
by  maintenance,  the  designer  should  consider  the  risks  of  field  maintenance  and 
minimize  the  characteristics  of  the  design  that  are  susceptible  to  operationally  induced 
reliability  deterioration.  Equally  important,  reliability  predictions  should  be  made  on 
realistic  operational  projections  for  degradation.  (DSMC,  “Designing  Quality  Into 
Defense  Systems”,  p.  28) 

However,  it  can  be  argued  that  reliability  requirements  can  and  should  be 
established  for  each  phase  or  product  life  cycle  of  a  system  such  as  s  torage, 
transportation,  installation,  standby,  and  operation.  Therefore,  a  realistic  reliability 
requirement  must  account  for  all  application  environments  and  the  time  proportions 
expected  in  each,  and  an  apportionment  of  the  requirement  across  the  life  cycle  phases 
accounts  for  deterioration  in  each  phase  (Reliability,  Maintainability,  and  Supportability 
Guidebook,  SAE,  p.  75).  Ultimately,  perhaps  contractors  should  attempt  to  account  for 
all  elements  contributing  to  the  combined  failure  rate  (Table  2.1)  and  provide  the 
government  with  a  confidence  interval  for  a  predetermined  readiness  performance  in  the 
form  of  operational  availability.  Such  ideas  are  open  to  dispute  and  will  be  discussed  in 
the  upcoming  analysis  chapter  of  this  thesis. 
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Figure  3.4.  Sustaining  Reliability  in  Production  and  Service.  (From:  DSMC, 
“Designing  Quality  into  Defense  Systems”,  p.  28) 

D.  CHAPTER  SUMMARY 

Beginning  with  the  initial  requirements  generation,  through  each  iteration  of  the 
systems  engineering  process,  and  ultimately  during  post -production,  reliability  must  be 
planned  for,  monitored,  accessed,  and  improved  during  the  maturation  of  a  weapon 
system.  The  greatest  impact  on  life  cycle  cost  and  future  operational  availability  are 
realized  during  the  early  phases  of  system  design  and  development,  and  thus,  logistics 
and  the  design  for  supportability  must  be  inherent  within  early  system  design 
development  if  the  results  are  to  be  cost-effective.  The  Department  of  Defense  (DoD) 
must  continue  to  strive  for  the  integration  of  acquisition  and  logistics  in  an  effort  to 
ensure  a  superior  product  support  process  by  focusing  on  total  ownership  cost, 
supportability  as  a  key  design  and  performance  factor,  and  logistics  emphasis  in  the 
systems  engineering  process  (DSMC  Acquisition  Chart,  2001).  Reliability  must  be  the 
focus  of  such  core  planning.  Fortunately,  as  discussed  in  this  chapter,  many  opportunities 
exist  throughout  all  phases  of  the  acquisition  process  to  effectively  address  reliability. 

The  next  chapter  examines  reliability  management  techniques  and  methodologies 

utilized  by  program  management  offices  as  well  as  common  issues  and  inhibitors 
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associated  with  reliability  management.  The  data  was  collected  via  an  electronic  survey 
and  the  results  are  presented  in  aggregate  form,  organized  by  general  themes. 
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IV.  MANAGING  RELIABILITY  IN  ACQUISITION  PROGRAMS 

A.  INTRODUCTION 

This  chapter  explains  the  methodology  utilized  for  data  collection  and  presents 
the  data  gathered  to  address  the  primary  and  subsidiary  research  questions.  The  data 
collected  relates  to  a  variety  of  technical,  programmatic,  managerial,  and  procedural 
issues  and  concerns;  common  practices;  and  acquisition  experiences  that  relate  to 
reliability.  The  data  presented  reflects  the  actions,  experiences,  and  perceptions  of  the 
acquisition  workforce  that  deals  with  reliability  management  issues  within  the  Marine 
Corps.  The  primary  source  of  data  collection  was  a  web -based  reliability  performance 
survey  that  was  distributed  to  targeted  program  management  offices  via  the  Acquisition 
Logistics  Office  at  Marine  Corps  Systems  Command.  The  survey  was  a  modification  of 
a  similar  survey,  previously  distributed  to  a  Program  Executive  Office  w  ithin  the  Army 
acquisition  community,  as  well  as  from  the  literature  review  and  the  background  research 
on  reliability,  described  in  Chapters  II  and  III.  A  copy  of  the  survey  can  be  found  in 
Appendix  B.  It  should  be  noted  that  the  survey  data  from  the  responding  program  offices 
is  presented  in  aggregate  form,  organized  by  general  themes,  and  summarized  in  tables 
created  by  the  author. 

B.  METHODOLOGY 

In  an  effort  to  determine  the  current  environment  for  reliability  management 
within  the  Marine  Corps  acquisition  community,  the  researcher  administered  an 
electronic  survey  to  various  personnel  within  the  Program  Offices  of  specific 
critical/pacing  end  items.  The  survey  directions  requested  attention  be  given  by  upper 
level  management  personnel  such  as  the  RVI  or  deputy  PM.  Respondents  included 
Program  Managers,  Program/Project  Team  Leaders,  reliability  engineers,  and  heads  of 
the  logistics  engineering  divisions.  The  questions  posed  were  intended  to  emphasize  the 
perspective  of  program  management  leadership  on  the  varied  tasks  involved  with 
reliability  management.  In  addition  to  the  qualitative -natured  questions  concerning 
management  and  procedural  issues,  numerous  quantitative  questions  were  included  to 
determine  and  compare  required  reliability,  estimated  or  predicted  reliability,  and 
achieved  reliability.  As  a  supplemental  source  to  gain  insight  into  reliability  management 


67 


issues,  interviews  were  also  conducted  with  current  acquisition  professionals  familiar 
with  program  and  reliability  management,  to  include  personnel  from  various  program 
offices,  the  test  community,  reliability  management  disciplines,  various  studies  and 
analyses  branches,  and  personnel  from  academic  disciplines. 

1.  Program  Demographics 

The  systems  originally  intended  for  research  were  limited  to  mature 
critical/pacing  end  items  included  in  the  Quarterly  Readiness  Reports  to  Congress  (M1A1 
tank,  AAV  family  of  vehicles,  LAV  family  of  vehicles,  5- ton  truck  family  of  vehicles, 
HMMWV  family  of  vehicles,  MK-48  LVS  Power  Unit,  and  M198  Howitzer).  All  the 
programs  are  Acquisition  Category  (ACAT)  level  I  and  are  part  of  the  Marine  Corps 
ground  equipment  inventory.  However,  it  should  be  noted  that  some  of  the  systems  were 
procured  with  the  Army  acting  as  the  executive  agent. 

Legacy  systems,  as  opposed  to  systems  in  development  or  recently  fielded 
systems,  were  targeted  due  to  the  expected  availability  of  achieved  field  reliability  data, 
which  was  to  be  compared  with  required  and  estimated  reliability.  Due  to  the  operational 
age  of  the  systems,  replacement  systems  are  currently  in  development  for  several  of  the 
systems. 

As  a  result  of  non-participation  by  a  significant  portion  of  the  targeted  programs, 
additional  willing  participants,  largely  from  the  AAAV  program  office,  offered  input  to 
the  qualitative  portion  of  the  survey.  However,  due  to  the  early  stage  of  the  AAAV 
development,  the  quantitative  data  questions  on  estimated  and  achieved  field  reliability 
were  not  applicable  to  the  program. 

2.  General  Survey  Question  Themes 

The  research  was  intended  to  evaluate  how  weapon  system  reliability 
performance  is  managed  throughout  the  acquisition  process  by  identifying  common 
inhibitors  and  enablers  of  effective  reliability  management,  why  they  occur,  lessons 
learned,  and  potential  methods  for  mitigating  the  inherent  risks.  To  do  so,  the  survey 
consisted  of  37  primary  questions,  some  with  subparts,  which  focused  on  five  major 
themes,  developed  for  the  purpose  of  this  thesis,  and  listed  below: 

•  management  approach  to  reliability 

•  determining  and  documenting  reliability  requirements 
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•  contracting  and  incentivizing  for  reliability 

•  reliability  testing 

•  comparing  and  assessing  required,  estimated,  and  achieved  field  reliability 

Collectively,  these  themes  correspond  to  issues  addressed  in  the  thesis  research 

questions. 

3.  Data  Presentation 

In  an  effort  to  obtain  disclosure  of  all  issues  associated  with  reliability 
management,  respondents  were  permitted  to  provide  information  under  the  premise  of 
non-attribution.  Likewise,  survey  instructions  specific  ally  stated  that  all  program 
responses  would  be  presented  in  aggregate  form.  Responses  were  received  from  only 
three  of  the  seven  programs  originally  solicited  for  participation.  As  a  result,  the 
researcher  sought  additional  programs  for  participation  to  gain  further  perspectives  on 
reliability  management  issues.  The  additional  programs  were  incorporated  through  their 
survey  responses,  interviews,  and  email  correspondence. 

The  subsequent  sections  provide  the  data  for  this  research  and  serve  as  the  basis 
for  analysis  in  Chapter  V.  The  survey  responses  and  corresponding  data  are  organized 
into  the  previously  mentioned  five  major  themes. 

While  all  the  themes  have  subparts,  each  theme  is  generally  presented  in  the  same 
fashion.  First,  the  purpose  of  the  survey  questions  within  that  main  theme  is  addressed. 
Next,  narrative  summaries  of  responses,  data  tables,  illustrative  examples  of  reliability 
management  experiences,  responses  in  the  form  of  quotes,  or  a  combination  of  such,  are 
presented.  Lastly,  the  author  summarizes  the  responses  and  data  to  exemplify  challenges 
that  program  managers  face  when  dealing  with  reliability  issues  of  their  systems. 

C.  MANAGEMENT  APPROACH  TO  RELIABILITY 

Purpose  of  Theme:  The  first  series  of  survey  questions  focus  on  how  reliability 
and  its  associated  risks  are  managed.  The  questions  asked  the  program  offices:  1)  what 
they  perceived  to  be  the  key  factors  that  contribute  to  reliability  problems  in  a  program; 
2)  how  reliability  performance  is  managed  within  a  PMO  in  terms  of  roles  and 
responsibilities,  documentation,  and  activities  utilized  to  recognize  and  evaluate  potential 
system  failures;  and  3)  their  opinion  and  understanding  of  the  amount  and  adequacy  of 

DoD  and  USMC  policy  and  guidance  on  reliability. 
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1.  Key  Factors  Contributing  to  Reliability  Performance  Problems 

Given  a  list  of  fifteen  common  prevailing  issues,  the  survey  participants  were 
asked  to  rank  order  what  they  perceived  to  be  the  top  five  factors  that  contributed  to 
reliability  problems  in  program  management.  Respondents  were  also  given  the 
opportunity  to  nominate  “other  issues”  and  rank  them  relative  to  the  fifteen  issues 
provided  in  the  survey.  Table  4.1  provides  a  compilation  of  the  top  responses,  presented 
in  an  overall  composite  order  of  merit  ranking,  from  the  most  significant  factor  to  the 
least  significant. 

Survey  Responses: 

TOP  RATED  FACTORS  CONTRIBUTING  TO 
RELIABILITY  MANAGEMENT  PROBLEMS 

1.  Traditional  test  &  evaluation  RAM  metrics  are  not  supported  by 
maintenance  data  sources  (unable  to  make  a  valid  comparison  b/n  RAM 
requirements  and  estimates  with  achieved  field  data) 

2.  Too  much  pressure  to  Held  systems  rapidly  (schedule  goals  outweigh 
reliability) 

3.  Need  more  qualified  reliability  personnel  in  PMOs 

4.  Unrealistic  reliability  requirements  with  inadequate  rationa  le 

5.  Poor  reliability  planning  and  growth  planning  (test  too  late) 

6.  Missing  or  poorly  written  ORD  reliability  requirements 

7.  Insufficient  reliability  testing  to  verify  requirements 

8.  Contractor  not  designing  for  reliability  sufficiently  above  the  requirement 

9.  Too  much  pressure  to  field  system  cheaper  (cost  goals  outweigh  reliability) 

10.  Not  consistently  improving  reliability  after  fielding 

11.  Inadequate  or  vague  policies  and  guidance  (need  updating) _ 

Table  4.1.  Top  Reliability  Management  Problems  as  Perceived  by  Survey 
Respondents  within  the  Acquisition  Community. 

Given  the  opportunity  to  nominate  their  own  factors  affecting  reliability 
management,  several  respondents  did  so,  providing  the  following  comments: 

•  PMs  are  not  provided  the  resources  or  authority  to  impact  reliability 

•  Engineers  pay  more  attention  to  meeting  performance  requirements  than 
to  reliability  requirements  when  they  should  be  considered  more  equally 

•  Traditionally,  PMs  have  been  evaluated  on  cost,  schedule,  and 
performance.  Thus,  reliability  often  got  pushed  aside 
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•  The  PMs,  the  Primes,  and  all  the  members  of  the  IPTs  should  be  evaluated 
on  readiness  performance  that  have  force  of  law 

•  Currently,  PMs  are  graded  on  cost  (testing  costs  and  costs  to  field), 
schedule,  and  performance  in  accordance  with  the  Defense  Acquisition 
Executive  Summary  (DAES)  vs.  SUPPORT  ABILITY,  in  the  form  of  LCC 
or  some  target  A, 

•  Dollars  drive  the  train  (acquisition  process)  instead  of  requirements 

Summary:  The  top  three  inhibitors  to  effective  reliability  management,  as  ranked 

by  the  survey  respondents,  were  clearly  identified  as  problematic  as  all  of  the  r  espondents 
chose  all  three  of  these  choices  as  one  of  their  top  five  ranked  issues.  Interestingly 
though,  twelve  of  the  fifteen  (survey  -provided)  choices  received  two  or  more  votes,  and 
five  of  the  fifteen  choices  received  at  least  one  vote  as  the  top  inhibitor  to  effective 
reliability  management. 

2.  Managing  Reliability  in  Acquisition  Programs 

The  next  series  of  questions  deal  with  program  management  approaches  to 
reliability,  to  include  the  perceived  roles  and  responsibilities  of  dealing  with  reliability, 
formal  documentation  of  a  reliability  program  plan,  and  activities  utilized  to  recognize 
and  evaluate  potential  system  failures. 

a.  Reliability  Roles  and  Responsibilities 

Survey  participants  were  asked,  “who  within  the  organization  was 
primarily  responsible  for  program  reliability  activities.”  The  author  desired  to  determine 
how  PMs  delegated  responsibility  for  reliability  activities  and  whether  there  was  a 
consistent  managerial  approach  in  doing  so.  If  the  respondents  indicated  that  reliability 
activities  were  conducted  within  the  context  of  an  Integrated  Product  Team  (IPT), 
responders  were  asked  if  the  IPT  was  formally  chartered. 

Survey  Responses :  Responses  varied  throughout  the  programs  without 
any  overwhelmingly  unified  response.  The  most  common  responses  indicated  that  either 
Logistics/Supportability  Team  Leaders  or  Project  Team  Leaders  had  been  delegated 
primary  responsibility  for  reliability  issues,  each  receiving  two  responses.  Only  the 
AAAV  program  indicated  the  use  of  reliability  IPTs.  Additionally,  two  programs 
recognized  that  the  PM  had  ultimate  responsibility  while  delegating  reliability  issues  to 
Team  Leaders  and  others.  Lastly,  one  program  respondent  could  not  identify  an 
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individual  or  team  that  had  overarching  responsibilities  associated  with  reliability 
activities,  choosing  the  “no  one  specifically”  survey  option,  possibly  suggesting  a  shared 
responsibility  amongst  multiple  sources. 


A  former  PM  commented  that  PMs  manage  by  exception  and  without  a 
specific  problem  or  issue,  reliability  and  the  other  engineering  disciplines  are  managed 
through  empowerment  of  technical  experts  (Masiello,  p.  39). 

Summary:  Responses  varied  as  to  the  individual  or  group  primarily 
responsible  for  program  reliability  activities,  and  responses  included  the  PM,  Reliability 
IPTs,  Logistics  Team  Leaders,  Project  Team  Leaders,  and  in  one  case,  no  one 
specifically.  No  survey  responses  indicated  that  primary  responsibility  for  reliability  fell 
upon  the  prime  contractor,  test  team  leader  or  testing  activities,  system  engineering  team 
leader,  or  the  Logistics  Management  Specialists  (LMS).  Overall,  PMs  seemed  to  rely 
upon  reliability  competency  outside  the  program  through  matrix  support. 

b.  Documenting  a  Program ’s  Reliability  Approach 

Survey  participants  were  asked,  “how  the  system  reliability  program  and 
the  corresponding  management  approach  were  formally  documented  within  the 
program(s).”  Choices  included:  reliability  program  plan,  contract  statement  of  work 
(SOW),  test  and  evaluation  master  plan  (TEMP),  single  acquisition  management  plan 
(SAMP),  no  formal  reliability  management  plan,  or  other. 

Survey  Responses :  Of  the  responses  received,  half  of  the  programs 
indicated  that  there  was  no  formal  reliability  program  plan.  One  respondent  noted,  “there 
is  no  requirement  for  PMs  to  have  a  formal  program  or  an  overarching  document 
describing  the  activities.”  Of  the  programs  which  had  a  reliability  plan,  most  relied  on 
the  contract  SOW  or  TEMP  to  address:  1)  how  they  intended  to  ensure  reliability  was 
treated  a  a  high  priority  objective,  2)  methodologies  and  plans  for  measuring  and 
achieving  reliability,  and  3)  the  resources  needed  to  execute  the  plan. 

c.  Activities/Tools  Used  to  Evaluate  Potential  Failures 

There  are  numerous  test  and  design  tools  available  to  program  offices  and 
contractors  that  help  to  ensure  that  reliability  is  “designed -in,”  early  in  the  program. 
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“Designing-in”  reliability  upfront  reduces  risk  and  is  less  costly  than  finding  design 
discrepancies  during  later  stages  of  testing,  evaluatio  n,  and  operational  use. 


By  inquiring  as  to  which  “activities  that  the  program(s)  implemented  to 
recognize  and  evaluate  potential  failures  and  causes,”  the  author’s  intent  was  to 
determine  the  risk  mitigation  techniques  which  programs  and  contractors  employed  to 
address  reliability  achievement.  The  survey  asked  participants  to  identify  all  the  testing, 
engineering,  and  other  technical  methods  used  in  their  respective  programs.  A  list  of 
fifteen  common  testing,  engineering,  and  other  technical  methods  and  techniques  used  to 
determine  and  evaluate  potential  failures  and  their  causes  was  provided  for  survey 
respondents  to  choose  from.  Additionally,  participants  were  provided  the  opportunity  to 
list  any  other  methods  utilized  by  their  programs. 

Survey  Responses:  As  expected,  developmental  and  operational  testing 
played  a  major  role  in  the  development  of  all  programs.  However,  the  extent  to  which 
programs  failed  to  utilize  other  reliability  risk  mitigation  techniques  to  determine 
reliability  achievement  was  rather  astounding.  There  was  one  outlier,  which  was  the  only 
system  examined  that  is  currently  in  development.  While  the  AAAV  program 
respondents  indicated  the  use  of  all  fifteen  techniques  listed,  the  other  program 
respondents  either  had  very  scarce  use  of  the  tools  or  were  not  aware  whether  the  original 
program  staffs  and  contractors  had  used  the  techniques. 

Each  program  indicated  that  it  utilized  only  one  of  the  following 
techniques:  Environmental  Stress  Screening  (ESS),  reliability  modeling,  FMECA, 
Reliability  Development/Growth  Test  (RD/GT),  or  FRACAS.  Additionally,  no  program 
indicated  the  use  of  reliability  allocation,  fault  tree  analysis,  probabilistic  failure 
assessment,  reliability  qualification  test,  PFMEA,  Weibull  analysis,  physics  of  failu  re 
(POF),  or  a  parts  control  program.  One  reliability  engineer  indicated,  “we  list  all  of  the 
tools  that  we  think  will  be  useful,  knowing  that  PMs  will  cut  many  of  them,  citing  fiscal 
constraints”  (Masiello,  p.  47). 

Summary:  Many  program  representatives  were  either  not  aware  of  the 
specific  techniques  utilized  to  ensure  reliability  was  “built  -in,”  or  the  original  staffs  did 
not  actually  use  the  available  tools.  However,  there  was  a  common  consensus  to  test 
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early  and  often,  and  use  knowledge  of  reliability  growth  to  implement  corrective  action. 
All  PMOs  reported  using  some  form  of  failure  analysis  as  an  integral  part  of  the  design 
process,  and  there  was  a  consensus  that  the  use  of  such  tools  that  incorporate  reliability 
prediction  and  achievement  into  system  design  was  beneficial. 

The  reader  should  be  reminded  that  in  most  cases,  the  survey  respondents 
were  not  the  original  PMO  staff,  and  the  respondents  may  be  aware  of  which  techniques 
were  utilized  only  by  reviewing  any  existing  documentation  th  at  was  retained  before  their 
arrival.  It  is  assumed  that  much  documentation  from  the  original  staff  or  the  contractor 
was  no  longer  available.  The  assumption  that  the  legacy  systems  did  not  take  advantage 
of  the  reliability  analysis  tools  may  be  invalid.  In  reality,  many  of  the  programs  may 
have  utilized  the  tools  more  than  indicated  in  the  survey,  and  the  respondents  were  not 
aware  of  the  previous  staffs’  or  contractors’  actions. 

3.  Existing  Policy,  Regulations,  and  Guidance  on  Reliability 

The  author  wanted  to  determine  the  level  of  existence  as  well  as  the  level  of 
awareness  of  reliability  policy,  regulations,  and  procedures.  Likewise,  the  author  desired 
the  opinions  of  the  acquisition  community  as  to  whether  the  existing  regulations  and 
guidance  were  sufficient  to  help  PMOs  manage  reliability  performance  in  their  programs. 
The  questions  posed  to  survey  participants  were,  “Are  you  aware  of  any  specific  DoD  or 
Marine  Corps  policy/regulations  regarding  weapon  system  reliability  management?  And, 
do  you  feel  that  existing  policy  and  regulations  on  reliability  provide  adequate 
guidance?” 

Survey  Responses:  Six  of  the  seven  respondents  that  chose  to  answer  this 
question  stated  they  were  unaware  or  unsure  of  any  policy  or  regulation  regarding 
reliability  management.  The  PM  that  answered  in  the  positive  did  not  cite  a  specific 
manual,  document,  handbook,  or  policy,  and  simply  stated  that  it  was  the  “program 
engineer’s  responsibility  (to  be  aware  of  this).”  Most  responses  and  interviews 
commented  on  frustration  concerning  the  lack  of  useful  documented  guidance. 
Additional  responses  are  paraphrased  or  quoted  below: 

•  lam  not  aware  of  any  policies  that  adequately  address  reliability 

•  You’ve  hit  the  nail  on  the  head  with  identifying  the  vague  nature  of  what 
is  currently  in  print 
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•  Due  to  acquisition  reform,  the  Government  has  steered  aware  from 
military  specs  and  standards.  Also,  this  makes  it  difficult  to  identify 
which  regulations  and  guidance  for  reliability  are  applicable  at  any  time. 

Summary:  According  to  the  responses,  the  acquisition  community  either  has  little 
guidance  or  is  not  aware  of  guidance  concerning  reliability  management.  Additionally, 
much  of  the  guidance  is  very  broad  scoped,  providing  little  detail  as  to  specific  reliability 
actions  to  be  taken  in  the  acquisition  process. 

D.  DETERMINING  AND  DOCUMENTING  RELIABILITY 
REQUIREMENTS 

Purpose  of  Theme:  The  next  group  of  questions  deals  with  reliability  in  the 
context  of  inputs  and  procedures  of  the  requirements  generation  process.  The  purpose  of 
the  questions  was  to  determine  and  assess  whether  a  reasonable  and  cooperative  process 
exists  between  the  Combat  Developer  and  the  Materiel  Developer,  if  reliability 
requirements  were  arbitrarily  set  or  not,  if  the  original  reliability  requirement  was 
documented,  and  if  so,  where  is  it  documented,  what  was  the  reliability  requirement,  and 
in  what  terms  was  it  identified. 

1.  Influencing  Realistic  Reliability  Requirements 

A  common  criticism  of  the  acquisition  process  is  that  system  requirements  are  not 
adequately  defined  or  are  often  unrealistic.  The  challenge  is  to  address  the  reliability 
requirements  in  terms  of  the  users’  operational  mission  needs  and  success  under  given 
conditions,  with  defined  mission  profiles,  environments,  and  durations  (Ryan,  p.  47). 

The  following  questions  and  corresponding  data  address  the  Materiel  Developer’s 
ability  to  influence  system  reliability  requirements,  and  the  level  and  terms  at  which  the 
requirements  were  set.  Participants  were  asked,  if  “the  PMO,  as  a  representative  of  the 
Materiel  Developer,  was  able  to  influence  incorporation  of  realistic  reliability 
requirements  into  the  process.”  They  were  also  asked,  “what  the  documented  reliability 
or  availability  requirement  was,  and  in  what  terms  it  was  measured  (i.e.,  MTBF,  MTBM, 
Ao,  MTBSA,  MTBOMF,  MTBEFF,  MTBOMA,  MTBMAT,  etc.)”. 

Survey  Responses:  In  nearly  all  cases,  materiel  developer  representatives  were 
able  to  provide  input  for  establishing  reliability  requirements. 
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Ability  to  Influence 

Percentage  of 

Reliability  Requirement 

Programs  Examined 

YES 

78  % 

NO 

0  % 

NOT  SURE 

22  % 

Table  4.2.  Influence  on  the  Requirements  Generation  Process. 


The  terms  in  which  reliability  requirements  were  identified  varied  from  program 
to  program.  Respondents  indicated  the  documentation  of  requirements  in  the  form  of: 
Mean  Miles  Between  Failure  (MMBF),  Mean  Miles  Between  Operational  Mission 
Failure  (MMBOMF),  Mean  Time  Between  Operational  Mission  Failure  (MTBOMF), 
Operational  Availability  (A,,),  Mean  Time  Between  Unscheduled  Maintenance 
(MTBUM),  and  Mean  Time  Between  Failure  (MTBF). 

Additional  related  responses  concerning  reliability  requirements  are  paraphrased 

below: 

•  Contractors  are  in  business  to  provide  the  Government  the  products  and 
services  we  request.  If  they  fail  to  do  so,  they  go  out  of  business.  The 
question  then  becomes,  are  we  asking  for  what  we  really  want  in  clear  and 
concise  terms?  When  a  program  fails,  too  many  people  in  this  business 
affix  blame  to  the  contractors.  Instead,  I  believe  that  the  Government  is 
ultimately  accountable  to  the  taxpayers.  Did  we  ask  for  what  we  needed? 
Did  we  select  the  right  contractor  to  do  the  job?  Did  we  provide  adequate 
support  and  oversight  to  the  project?  I  realize  that  very  few  officials  in 
Government  are  willing  to  ask  such  tough  questions. 

•  MCCDC  has  the  responsibility  of  creating  the  requirements,  but  the  PM 
office  comments  on  the  requirements  and  their  rational  with  MCCDC 

•  In  order  to  determine  user  reliability  requirements,  emphasis  must  be 
placed  on  understanding  the  user’s  system  readiness  and  mission 
performance  requirements;  and  translating  them  into  system  requirements 
that  can  be  designed,  implemented,  and  verified 

Summary:  According  to  the  responses,  it  appears  that  programs  actively 

participate  with  the  Combat  Developer  to  determine  the  requirements,  including  those 

requirements  relating  specifically  to  reliability  as  part  of  the  RAM  determination  process. 
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Thus,  it  may  be  assumed  that  a  reasonable  and  cooperative  process  exists  between  the 
Combat  Developer,  Materiel  Developer,  and  the  user  representative. 

A  review  of  the  terms  in  which  the  reliability  requirement  is  identified  varies  from 
program  to  program,  indicating  that  there  is  not  a  standard  operational  terminology  in 
which  reliability  must  be  expressed.  While  this  likely  allows  for  flexibility,  there  must  be 
an  agreement  and  understanding  between  the  Government  and  contractor  of  those  terms, 
as  further  sections  will  indicate. 

2.  Reliability  as  a  KPP  in  the  ORD 

As  the  survey  responses  indicated  in  the  previous  section,  most  programs  had 
documented  reliability  requirements,  while  identifying  the  specific  terms  (MTBF, 
MTBM,  etc.)  used  in  defining  requirement.  It  then  becomes  useful  to  discover  where  the 
requirements  are  documented. 

The  author  hoped  to  ascertain  the  relative  importance  of  reliability  with  respect  to 
other  performance  parameters.  Participants  were  asked  if  reliability  requirements  were 
identified  as  Key  Performance  Parameters  (KPP)  in  the  Operational  Requirements 
Document  (ORD).  Additionally,  they  were  asked  whether  the  requirement  was  in  an 
objective  and  quantifiable  form  that  contrac  tors  and  the  Government  could  easily  agree 
upon. 

Survey  Responses :  Only  two  responses  indicated  the  use  of  a  reliability 
requirement  as  a  KPP  -  one  of  which  had  a  sister  service  as  the  executive  agent,  and  the 
other  was  the  AAAV,  which  is  the  only  system  still  under  development  from  which 
survey  responses  were  collected.  Meanwhile,  none  of  the  remaining  legacy  programs 
examined  included  reliability  as  a  KPP.  Some  responses  indicated  that  their  current 
program  staff  could  not  locate  the  ORD  due  to  the  time  that  has  passed  and  the  turnover 
of  personnel.  Additional  related  responses  were: 

•  RAM  requirements  are  usually  not  KPPs.  So  when  there  are  cost  or 
schedule  overruns,  these  are  the  first  to  take  a  hit. 

•  Reliability  and  maintainability,  along  with  performance,  should  act  as 
equal  partners  in  the  requirements  generation  process 

•  Test  to  requirements  in  the  ORD.  If  reliability  is  not  a  KPP  in  the  ORD,  it 
gets  pushed  aside  due  to  other  requirements  precedence 
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•  There  seems  to  be  a  huge  traceability  problem.  We  couldn’t  even  find  the 
(undisclosed  program  name)  ORD  until  (undisclosed  analyst)  called  an  old 
friend  at  the  contractor  who  had  kept  a  copy. 

Summary:  While  reliability  requirements  were  typically  not  identified  as  KPPs, 
programs  agreed  that  reliability  was  an  important  priority  that  received  varying  degrees 
of  attention. 

As  previously  mentioned,  all  of  the  systems  examined  were  legacy  systems  with 
the  exception  of  the  AAAV.  Interestingly,  the  newest  system  examined,  which  is  still 
under  development,  has  designated  reliability  as  a  KPP.  In  fact,  the  AAAV  has  a  very 
specific  MTBOMF  threshold  as  a  KPP  for  the  Milestone  C  decision.  Additionally,  to 
ensure  that  the  requirement  is  in  an  objective  and  quantifiable  term  that  the  contractor  and 
the  Government  can  agree  upon,  the  AAAV  contractor  was  “given  the  Failure  Definition 
and  Scoring  Criteria  which  was  the  basis  of  determining  whether  a  failure  was  an 
operational  mission  failure.” 

E  CONTRACTING  AND  INCENTTVIZING  FOR  REL IABILITY 

Purpose  of  Theme:  The  questions  and  corresponding  survey  responses  in  this 
section  relate  to  the  role  of  reliability  in  the  source  selection  and  contracting  process.  The 
overall  intent  of  this  series  of  questions  was  to  determine  how  and  to  what  extent 
reliability  requirements  were  developed  into  contractual  agreements. 

1.  Reliability  as  a  Source  Selection  Factor 

Programs  were  queried  as  to  whether  reliability  was  included  as  a  factor  in  source 
selection. 

Survey  Responses  :  With  the  exception  of  the  AAAV,  the  program  respondents 
replied  that  either  reliability  was  not  a  factor  in  source  selection  or  they  were  not  certain 
if  reliability  was  a  factor  in  source  selection  due  to  the  time  that  had  passed  since  the 
program  was  originally  contracted  and  the  lack  of  documentation  in  the  PM  offices. 

Summary:  While  reliability  was  not  a  factor  in  source  selection  for  the  legacy 
systems  examined,  some  respondents  gave  the  impression  best  put  by  one  individual, 
“Reliability,  with  its  impact  on  O&S  costs,  should  receive  critical  attention  in  the  market 
investigation,  solicitation,  and  source  selection  process.  Unfortunately,  I  believe  this  is 
typically  not  the  case.” 
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2.  Reliability  Requirements  in  Contracts 

The  second  contract  related  question  inquired  as  to  how  operational  reliability 
requirements  in  the  ORD  were  translated  into  contractual  requirements. 

Survey  Responses:  Roughly  two-thirds  of  the  participating  survey  respondents 
indicated  the  ORD  paragraphs  relative  to  reliability  were  restated  in  the  Statement  of 
Work  or  performance  specifications,  indicating  that  the  contract  requirement  was  very 
similar  to  the  ORD  requirement.  One  of  the  oldest  systems,  which  has  exceeded  its 
intended  life  cycle  by  over  a  decade  and  a  half  due  to  extensive  upgrades  and  Depot 
Level  Maintenance,  indicated  that  comprehensive  reliability  requirements  were  not 
adequately  stated  in  the  original  contract.  Conversely,  the  AAAV  sets  precedence  for 
future  systems  by  applying  “additional  levels  of  reliability  to  the  contract  as  the 
performance  specifications  (in  the  contract)  set  the  bar  a  little  higher  than  the  ORD.” 
Additional  related  responses  concerning  the  contractual  reliability  requirements  are 
provided: 

•  While  predicted  reliability  typically  comes  form  contractor  claims,  we 
need  some  confidence  level  that  it  is  a  good  system  of  merit  for  predicted 
reliability.  We  must  contract  with  the  Prime  (contractor)  for  a 
commitment  to  some  target  A,,. 

•  We  need  to  make  readiness  targets  contract  items 

•  It  would  require  contract  changes  to  hold  contractors  accountable  to  their 
estimates 

•  Reliability  objectives  should  be  translated  into  quantifiable  and  verifiable 
contractual  terms  and  allocated  through  the  system  design  hierarchy 

•  Contractual  requirements  should  be  traceable  to  operational  requirements 
and  capable  of  verification 

•  We  should  adopt  the  null  hypothesis  that  states  the  MTBF  is  not  what  the 
contractor  claims,  but  rather  what  the  contractor  proves 

Summary:  In  terms  of  translating  user  operational  requirements  to  contractual 
requirements,  all  but  one  of  the  legacy  systems  examined  indicated  that  the  contractual 
requirement  was  very  similar  to  the  ORD  requirement,  and  ORD  paragraphs  relative  to 
reliability  were  simply  restated  in  the  SOW  or  specifications.  The  remaining  legacy 
system  stated  that  the  comprehensiv  e  reliability  requirements  were  not  adequately  stated 
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in  the  original  contract.  Conversely,  the  AAAV  applied  additional  levels  of  reliability  to 
the  contract. 

3.  Contracting  Incentives  for  Reliability 

The  use  of  meaningful  contract  incentives  for  achievin  g  predetermined  reliability 
performance  is  a  method  to  encourage  contractors.  Survey  participants  were  questioned 
if  incentives  that  are  specifically  tied  to  achieving  system  reliability  performance 
requirements  were  employed  in  their  programs’  contracts,  and  if  so,  did  the  incentives 
achieve  their  desired  effects. 

Survey  Responses :  The  respondents  representing  the  legacy  systems  indicated 
that  contract  incentives  were  not  utilized  for  the  original  purchases  of  their  systems. 
However,  the  AAAV  program  staff  cited  the  use  of  a  Cost  Plus  Award  Fee  (CPAF) 
contract,  and  further  indicated  that  reliability  has  been  used  as  one  of  the  award  fee 
criterion  on  numerous  occasions  thus  far.  Additional  comments  are  provided  below: 

•  If  a  contractor  does  not  meet  the  predetermined  reliability  goals,  it  should 
be  considered  a  latent  defect 

•  We  must  tie  the  contractor  to  LCC  through  reliability.  In  other  words,  we 
must  reduce  life  cycle  support  costs  through  reliability  warranties  and 
incentives 

•  Incentives  should  be  created  to  reward  for  good  systems  in  terms  of 
logistics 

Summary:  Of  the  legacy  systems  examined,  there  was  no  apparent  use  of  contract 
incentives  for  reliability  achievement. 

F.  RELIABILITY  TES  TING 

Purpose  of  Theme:  Test  and  evaluation  activities  are  a  critical  part  of  every 
program  as  they  serve  to  aid  in  the  development  of  a  system  and  to  verify  that  the  system 
meets  specified  standards.  The  questions  in  the  proceeding  sections  are  concerned  with: 
1)  the  adequacy  of  time  and  funding  allotted  for  reliability  testing  during  developmental 
testing,  2)  general  agreement  and  common  understanding  on  measures  to  determine 
reliability  performance  during  testing,  and  3)  the  use  of  IOT&E  entrance  criteria  relative 
to  reliability. 
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1.  Resources:  Time  and  Funding  Constraints 

In  an  attempt  to  achieve  program  objectives,  program  management  requires 
making  trade-offs  in  terms  of  cost,  schedule,  performance,  and  supportability.  Programs 
were  queried  as  to  whether  the  amount  of  time  and  funding  allotted  for  reliability  testing 
during  DT&E  was  sufficient. 

Survey  Responses  :  All  but  one  program  indicated  an  insufficient  amount  of  time 
and  funding  allotted  for  reliability  testing  during  developmental  testing.  However,  this  is 
not  surprising  in  the  acquisition  world  wher  e  program  offices  continuously  are  forced  to 
conduct  trade-offs.  One  program  summarized  the  constrained  resource  situation  by 
stating,  “there  is  never  enough  time  or  money  because  the  more  time  and  money 
(available),  the  more  failures  that  can  be  uncovered  and  corrected.” 

Summary:  A  common  perspective  relayed  by  the  program  offices  is  a  lack  of 
time  and  money  to  conduct  adequate  levels  of  reliability  testing  which  are  needed  to 
achieve  a  substantial  confidence  level  of  the  system  reliability.  In  fact,  data  from  the  first 
survey  question  indicated  that  too  much  pressure  to  field  systems  quickly  and  too  much 
pressure  to  field  systems  cheaply  were  respectively  the  second  and  ninth  ranked 
inhibitors  to  effective  reliability  management. 

2.  Agreement  on  Reliability  Measures  for  Tests 

The  concept  of  reliability  is  often  used  without  precise  definition,  while  the 
terminology  is  non-standard  throughout  the  logistics  community  and  tends  to  depend  on 
the  system  being  developed.  However,  while  creating  DoD  requirements  documentation, 
contract  specifications,  and  test  documentation,  it  is  very  important  that  all  main  concepts 
are  addressed  in  an  unambiguous  way  so  that  all  parties  involved  (to  include  the  user, 
combat  developer,  materiel  developer,  PM,  contractor,  and  tester)  understand  the  terms. 
Survey  participants  were  asked  if  the  user,  contractor,  tester,  and  PM  all  agreed  upon  the 
method  used  to  determine  reliability  performance  during  testing. 

Survey  Responses :  One  of  the  legacy  programs  answered  affirmatively,  stating 
that  the  agreed  upon  method  could  be  found  in  the  TEMP.  The  remaining  systems 
indicated  that  they  were  uncertain  if  such  an  agreement  had  been  made  amongst  all 
parties.  The  numerous  “not  certain”  responses  are  likely  a  result  of  the  time  that  had 


81 


lapsed,  the  turnover  of  personnel,  and  the  loss  of  documentation  since  the  test  phases  had 
occurred  years  prior. 

3.  Testing  to  Determine  Reliability  Requirements  Conformance 

a.  Initial  Operational  Test  &  Evaluation  (IOT&E)  Entrance 
Criteria 

Operational  Test  and  Evaluation  is  the  final  test  conducted  prior  to  the 
decision  on  whether  the  system  will  proceed  with  full  rate  production.  Given  the 
significance  of  this  program  gate,  entrance  criteria,  relative  to  reliability,  is  often 
established  to  ensure  that  the  system  is  prepared  for  Initial  Operational  Test  and 
Evaluation.  Meeting  reliability  entrance  criteria  commonly  involves  testing  reliability  in 
Developmental  Testing  activities  and  involves  validating  required  reliability  levels.  Such 
entrance  criteria  are  most  often  required  by  the  independent  testing  organization. 

Survey  participants  were  asked  if  their  respective  programs  had  specific 
IOT&E  criteria  relative  to  reliability. 

Survey  Responses  :  A  significant  majority  of  the  responding  programs  had 
IOT&E  entrance  criteria  relative  to  reliability.  Furthermore,  most  criteria  were 
established  in  very  specific  terms,  such  as  MTBOMF.  Notably,  the  AAAV  program 
indicated  that  the  Milestone  C  criteria,  which  allows  the  program  to  build  Low  Rate 
Initial  Production  (LRIP)  vehicles  for  IOT&E,  was  to  “demonstrate  system  reliability 
within  the  Growth  Curve  at  80%  confidence  through  a  mix  of  test  data  and  analysis.”  On 
the  other  hand,  one  program  indicated  that  IOT&E  entrance  criteria  were  not  used  by  the 
sister  service  executive  agent,  and  one  program  was  uncertain  if  entrance  criteria  were 
utilized. 

b.  Reliability  Demonstration  During  Developmental  and 
Operational  Testing 

Programs  were  asked  to  what  level  were  their  systems’  ORD  reliability 
requirements  demonstrated  during  developmental  testing,  operational  testing,  and  during 
sustainment.  Additionally,  programs  were  asked  to  what  level  were  the  contractors’ 
reliability  estimates  demonstrated  during  developmental  testing,  operational  testing,  and 
sustainment.  By  posing  such  questions,  the  author  intended  to  gain  a  better 
understanding  of  the  required  level  of  reliability  demonstration  prior  to  the  program 
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proceeding,  whether  reliability  requirements  and  contractor  estimates  were  sufficiently 
realistic  to  be  achieved,  and  whether  there  was  a  correlation  between  success/failure  in 
DT&E,  OT&E,  and  sustaiment  with  respect  to  reliability  performance.  Survey  test 
related  responses  are  provided  below: 

•  We’ve  found  they  (PMOs)  don’t  have  much  documentation  from  the 
DT&E  phase  of  programs.  We’ve  had  more  success  finding  OT&E 
reports  from  MCOTEA  since  these  are  Government-run  and  usually 
archived  by  DTIC  or  service  libraries. 

•  The  DT&E  reports  were  probably  submitted  to  PM  staff  and  ended  up 
taken  or  lost  when  those  people  moved  on 

•  It  is  during  T&E  that  we  (Government)  must  validate  the  contractor’s 
estimates.  Under-  or  over-estimating  reliability  will  cause  limited  funds  to 
be  allocated  unwisely 

•  Once  the  contractor  submits  their  RAM  estimate,  it  becomes  the 
Government’s  estimate  if  we  accept  it.  Therefore,  it’s  up  to  us  to  become 
involved  in  this  process  and  to  conduct  independent  testing  as  necessary  to 
verify  such  estimates. 

•  Demonstration  of  required  reliability  performance  levels  prior  to  system 
fielding  is  a  challenge 

•  Estimated  or  measured  reliability  should  be  used  to  evaluate  the  design 

•  Achievement  of  contractual  requirements  should  be  verified  through  a 
combination  of  engineering  analysis  and  test  results 

•  Test  to  (the)  requirements  in  the  ORD 

•  Determination  of  contractual  compliance  based  on  engineering  analysis 
without  supporting  test  data  can  lead  to  erroneous  conclusions 

•  We  must  conduct  Logistics  Test  &  Evaluation  (LT&E)  early  and 
throughout  the  DT&E,  allowing  program  office  personnel  to  determine 
what  needs  to  be  adjusted  to  provide  the  required  system  support 
throughout  the  program’s  life  cycle 

Survey  Results :  Qualitative  responses  to  this  question  were  limited.  None 
of  the  participating  programs  were  able  to  identify  the  level  to  which  the  contractors’ 
estimates  were  demonstrated  (during  testing  phases  or  sustainment)  due  to  the  fact  that 
contractor  estimates  were  not  retained. 

Additionally,  in  some  cases,  the  ORD  could  not  be  found,  meaning  the 
original  reliability  requirement  was  not  retained  either. 
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Lastly,  most  programs  indicated  that  achieved  field  reliability  data 
compiled  during  the  sustainment  phases  of  the  respective  systems  is  suspect  to  error, 
making  the  comparison  of  such  data  with  the  contractor  estimates  and  original  ORD 
requirements  (when  available)  questionable.  The  proceeding  section  is  devoted  to  such 
issues  and  concerns. 

G.  COMPARING  AND  ASSESSING  REQUIRED,  ESTIMATED,  AND 

ACHIEVED  RELIABILITY 

Purpose  of  Theme :  Programs  that  have  carefully  planned  and  executed  reliability 
management  techniques  will  benefit,  in  the  way  of  decreased  life  cycle  costs  and 
increased  operational  availability,  during  sustainment  of  the  system.  However,  reliability 
performance  of  a  system  should  be  continually  assessed  throughout  its  lifecycle. 
Programs  often  assume  reliability  to  be  what  the  contractor  states  it  to  be  instead  of 
determining  progress  and  compliance  with  reliability  estimates  and  requirements 
throughout  the  lifecycle  sustainment  phase. 

This  section  summarizes  previous  questions  by  compiling  and  comparing  survey 
responses  concerning  ORD  reliability  requirements,  contractor  reliability  estimates,  and 
achieved  reliability.  The  data  is  intended  to  determine  whether  PMOs  and  the  logistics 
community  are  adequately  engaged  in  tracking  and  improving  system  reliability  through 
a  systematic  process  of  collecting  reliability  trend  data. 

1.  Maintaining  Original  Contractor  Estimates 

Participants  were  queried  as  to  whether  or  not  the  contractor  reliability  estimate 
was  documented,  and  if  so,  where  was  it  documented,  who  retains  the  documentation, 
what  was  the  estimate,  and  in  what  terms  was  it  measured  (i.e.,  MTBF,  MTBM, 
MTBOMF,  MMBF,  A,,  etc.). 

Survey  Responses :  All  responding  programs  indicated  that  either  the  contractor 
reliability  estimates  were  not  documented  or  the  documentation  was  not  retained.  One 
respondent  stated  that  the  estimates  were  not  retained  because  the  “program  was  too  old”. 
Other  related  responses  are  provided: 

•  I  highly  suspect  you  will  have  to  go  to  each  PM’s  office  in  an  attempt  to 
locate  this  information  (ORD,  contract,  reliability  rqmt,  reliability 
estimate,  and  achieved  reliability) 
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•  Where  will  you  get  copies  of  the  contractors’  RAM  estimates?  Does 
anyone  retain  these  documents  after  the  system  is  fielded? 

•  Once  the  contractor  submits  their  RAM  estimate,  it  becomes  the 
Government’s  estimate  if  we  accept  it. 

•  ...  we  (the  Government)  must  validate  the  contractor’s  (reliability) 
estimate. 

•  Unfortunately,  there  is  not  an  annual  inspection  that  checks  whether  or  not 
they  are  maintaining  the  data  and  information. 

2.  Collection  and  Computation  of  Achieved  Field  Reliability 

Calculation  of  system  reliability  that  is  being  achieved  in  the  field  is  necessary  in 
order  to  determine  whether  the  mean  time  between  failures  is  increasing,  decreasing,  or 
remaining  constant  with  age.  In  other  words,  such  data  analysis  and  assessment  of 
reliability  performance  help  to  determine  if  equipment  is  in  the  “wear  -out”  phase  of  its 
life  cycle  and  at  the  end  of  its  economic  useful  life. 

Participating  program  representatives  were  asked  whether  the  achieved  reliability 
of  their  programs  had  been  computed,  and  if  so,  what  was  the  overall  achieved  reliability 
and  in  what  form  was  it  calculated.  It  is  important  to  recall  from  Chapter  II  that  there  are 
numerous  forms  of  measurements  that  relate  to  reliability  achievement,  to  include  MTBF, 
MTBM,  MTBOMF,  MMBF,  A.  and  others.  Table  4.3,  in  the  next  section  provides  the 
results.  Additionally,  various  personnel  provided  the  remarks  below: 

•  USMC  maintenance  management  systems  record  maintenance  activities 
and  not  system  failures,  (which  are  required  to  calculate  MTBF),  while  the 
systems  do  not  distinguish  between  preventative  and  corrective 
maintenance  activities 

•  Rather  than  computing  ‘failure  rates,’  we  are  focusing  instead  on 
‘maintenance  rates’  because  our  AIS  systems  do  not  directly  record 
failures;  MIMMS/ATLASS  records  maintenance  events.  We  believe  that 
maintenance  rate  analysis  is  a  feasible  surrogate  for  traditional  failure  rate 
analysis;  based  on  existing  data  sources,  that’s  probably  the  best  we  can 
do.  Even  then,  we’ve  run  into  significant  obstacles  and  data  quality  issues 
with  MIMMS. 

•  Also,  any  metric  based  on  operational  usage  data  won’t  work.  The  meter 
readings  in  MIMMS  are  inconsistent  and  inaccurate  (e.g.,  odometer 
reading  =  999999,  mixing  engine  hours  and  odometer  miles,  etc.). 

•  Utilization  data  is  not  consistent 

•  It  is  hard  to  distinguish  between  corrective  and  preventive  maintenance 
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Summary:  Lack  of  necessary  data  and  related  weaknesses  in  the  Marine  Corps 
maintenance  management  data  systems  prevent  the  calculation  of  MTBF.  Specifically, 
operational  usage  data  is  required  to  calculate  failure  rate,  a  key  indicator  of  reliability 
performance.  Thus,  the  maintenance  rate  has  often  been  used  as  a  substitute  for  failure 
rate,  but  when  utilizing  MTBM  in  place  of  MTBF,  it  is  important  to  recall  that  MTBM 
does  not  distinguish  between  preventive  and  corrective  maintenance  activities. 
Furthermore,  issues  with  data  quality  derived  from  maintenance  management  systems 
also  make  the  calculation  of  MTBM  skeptical.  Thus,  many  professionals  believe  that  the 
next  best  calculation  is  the  R-rating  or  readiness  rating,  known  as  the  SORTS  equipment 
condition  rating,  calculated  as  follows  as  per  MCBul  3000: 

QtyPossessed  —QtyN otMissionCapable 
QtyPossessed 


The  R-rating  basically  provides  a  snapshot  of  operational  availability,  for  which 
the  calculation  is  shown  below: 


uptime  _  MTBM 
uptime  +  dowtime  MTBM  +MDT 


UPTIME 

_ OT+ST _ 

OT  +ST+ALDT  +CMT  +PMT 


UPTIME 


DOWNTIME 


The  calculation  of  operational  availability  is  affected  by  inhibiting  factors  in  the 
logistics  system  such  as  administrative  delay  time,  order  ship  time  delays,  and  delays  in 
corrective  and  preventive  maintenance.  Additionally,  standby  time  (ST)  is  not  recorded 
and  distinguished  from  operating  time  (OT).  Lastly,  the  data  is  extracted  from  disparate 
sources  (MIMMS  and  ATLASS). 

3.  Comparing  Required  Reliability,  Estimated  Reliability,  and  Achieved 
Reliability 

It  is  important  to  have  a  systematic  process  in  place  for  collecting  and  comparing 
reliability  data  for  several  reasons.  The  Marine  Corps  must  be  able  to  calculate  and 
compare  the  reliability  that  is  being  achieved  in  the  field  during  post -production  with  the 
required  and  estimated  reliability  in  order  to  determine  contactor  compliance, 
successfully  hold  contractors  to  their  estimates,  and  determine  if  the  user  reliability 
requirement  is  met. 
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The  data  from  the  questions  in  the  previous  sections  were  combined  in  an  attempt 
to  determine  the  “reliability  gap”  of  the  legacy  systems,  or  the  difference  between  the 
required  reliability,  contrac  tor  reliability  estimates,  and  achieved  reliability. 
Additionally,  programs  were  asked  what  organization(s),  if  any,  have  compared  and 
assessed  actual  achieved  reliability  of  fielded  systems  to  the  original  requirements  and 
contractors’  estimates. 

Survey  Responses:  All  of  the  legacy  system  respondents  stated  that  no 
organization,  internal  or  external  to  their  programs,  has  formally  or  informally 
determined  the  “reliability  gap”  for  the  respective  systems  up  to  this  point  in  time.  One 
program  did  note  that  MCCDC  is  concurrently  conducting  a  similarly  related  study 
entitled  “Sustainment  Consequences  of  Acquisition  Decisions”,  which  is  sponsored  by 
MARCORSYSCOM. 

Table  4.3  provides  a  comparison  of  three  sample  programs’  reliability 
requirements,  contractor  reliability  estimates,  and  achieved  reliability,  as  well  as 
participants’  personal  perspectives  when  provided. 


Program  1 

Program  2 

Program  3 

Reliability 

Requirement 

43.5  MTBOMF 
threshold 

600  MMBF 
(not  delineated  in 
the  ORD) 

Not  Known 

Contractor 

Reliability 

Estimate 

“Estimate  not 

documented ” 

“Estimate  not 
retained ;  program 
is  too  old  ” 

“Don 't  know  if 
documented : 

not  retained ” 

(Post-Production) 

Achieved 

Reliability 

89  MTBOMF; 
“data  is  suspect" 

“Not  certain.  The 
Marine  Corps  does 
not  do  a  good  job 
capturing  this  data  ” 

“  >  80% 
Readiness  ” 

Table  4.3.  Reliability  Gap. 
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Additional  comments  concerning  the  computation  and  comparison  of  required, 
estimated,  and  achieved  reliability  are  provided  below: 

•  Traditional  T&E  RAM  metrics  are  not  supported  by  our  maintenance  data 
sources  so  how  will  you  make  a  valid  comparison  between  contractor 
RAM  estimates  and  actual  data? 

•  Our  experience  here  (Studies  and  Analy  sis  branch)  is  that  data  is  not  easy 
to  come  by 

•  Reliability  focus  should  not  end  with  OT&E.  Once  the  system  is  fielded, 
reliability  should  become  a  permanent  part  of  the  PM’s  and  Logistics 
Management  Specialist’s  (LMS)  sustainment  activities.  Critical  system 
and  components  must  be  identified  where  low  reliability  rates  are 
hampering  mission  accomplishment. 

•  Currently  using  the  wrong  metrics 

Summary :  It  is  well  known  that  “you  can’t  manage  what  you  don’t  measure,”  and 
in  general,  responses  indicate  that  there  is  a  lack  of  a  systematic  process  for  collecting 
reliability  trend  data  beyond  readiness  ratings.  Furthermore,  what  data  that  does  exist  is 
suspect  to  error  and  corruption  as  a  result  of  the  current  maintenance  management 
automated  information  systems. 

There  was  also  a  consensus  that  traditional  test  and  evaluation  RAM  metrics  are 
not  supported  by  maintenance  management  data  systems.  One  may  recall  that  this  was 
voted  as  the  top  rated  inhibitor  to  effective  reliability  management,  according  to  the 
responses  from  the  first  question  of  the  survey. 

a.  MTBM  Computation 

A  study  was  completed  on  08  June  2002  by  Captain  Jake  Enholm  in  an 
attempt  to  formulate  a  methodology  for  determining  systemic  MTBM  and  equipment 
parts’  (NSNs)  failure  rates  using  current  warehoused  maintenance  management  data 
drawn  from  MIMMS/ATLASS  II/SASSY  data  fields.  In  the  calculations  used  in  this 
study,  MTBF  and  MTBM  periods  were  combined  as  the  model  used  both  preventive  and 
corrective  maintenance  actions  combined.  The  results  of  the  study  indicated  that  it  is 
sometimes  possible  to  calculate  a  sample  mean  or  median  time  between 
maintenance/failure  for  certain  equipment  in  the  Marine  Corps.  However,  the  study 
indicated  that  the  accuracy  of  the  analysis  is  suspect  to  weaknesses  in  the  Marine  Corps’ 
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maintenance  management  data  systems.  A  full  version  of  the  study  is  available  in 
Appendix  C. 

b.  Depot  Level  Maintenance  Program  Effects 

Due  to  the  age  of  the  respective  systems,  concerns  about  increasing  failure 
rates  and  increasing  costs  to  maintain,  and  in  some  cases,  declining  readiness  trends, 
some  of  the  systems  examined  have  undergone  Depot  Level  Maintenance  (DLM) 
programs  such  as  “Service  Life  Extension  Program”  (SLEP)  or  “Inspection  and  Repair 
Only  As  Necessary”  (IROAN).  Programs  such  as  these,  which  modify,  upgrade,  or 
change  the  designs  of  the  systems,  as  well  as  overhaul  major  components,  obviously 
affect  the  reliability  of  the  systems  and  skew  the  data  that  is  attempting  to  be  compared. 
Therefore,  to  take  this  factor  into  consideration,  survey  participants  were  specifically 
asked  whether  their  programs  had  undergone  and  type  of  Depot  Level  Maintenance 
programs  and  what  the  effect  was  on  reliability.  Answers  varied  greatly.  Some  programs 
that  had  undergone  DLM  programs  claimed  drastic  changes  in  reliability  performance 
while  others  claimed  there  was  no  significant  change  following  completion  of  the 
maintenance  activities.  Another  respondent  noted  that  his  program  was  scheduled  for 
mid-life  rebuild,  but  the  action  was  never  carried  out  due  to  funding  constraints. 

c.  Reliability  Growth  Programs 

Reliability  growth  is  the  improvement  in  a  reliability  parameter  over  a 
period  of  time  resulting  from  changes  in  product  design  or  the  manufacturing  process.  A 
structured  reliability  growth  program  is  typically  created  with  specific  interim  reliability 
goals  and  test  events.  As  the  system  design  matures,  testing  is  performed  at  designated 
intervals  to  identify  actual  or  potential  sources  of  failure. 

Managing  reliability  growth  requires  systematic  planning  for  reliability 
performance  achievement  as  a  function  of  time  and  other  resources.  This  involves 
controlling  the  ongoing  rate  of  achievement  by  reallocation  of  resources  based  on 
comparisons  required,  planned,  estimated,  and  ass  essed  reliability  values  (Ryan,  p.  42). 
Formal  reliability  growth  programs  serve  to  not  only  ascertain  requirement  compliance, 
but  to  also  identify  potential  problems  early  in  development. 

Survey  participants  were  asked  whether  their  programs  incorporated  a 

formal  reliability  growth  program.  With  the  exception  of  the  AAAV  program,  still  under 
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development,  none  of  the  programs  examined  had  a  formal  reliability  growth  program  in 
place.  However,  there  was  a  general  agreement  that  the  focus  should  be  on  disc  overing 
design  flaws  early  and  fixing  them  as  early  as  possible  to  avoid  potential  cost  overruns  in 
the  future  of  the  program. 

H.  CHAPTER  SUMMARY 

This  chapter  presents  the  methodology  used  and  the  data  gathered  from  the 
survey,  interviews,  and  emails.  Professionals  within  the  Marine  Corps  acquisition 
community  directly  contributed  by  providing  information  about  their  programs, 
experiences,  and  perspectives  with  respect  to  reliability  management.  The  data  was 
organized  into  five  major  themes:  1)  management  approach  to  reliability,  2)  determining 
and  documenting  reliability  requirements,  3)  contracting  and  incentivizing  for  reliability, 
4)  reliability  testing,  and  5)  comparing  and  assessing  required,  estimated,  and  achieved 
field  reliability. 

It  should  be  noted  that  the  survey  responses  were  from  individuals  that  inherited 
the  programs  long  after  the  systems  were  developed  and  fielded.  Consequently,  many  of 
the  responses  were  a  result  of  a  lack  of  documentation  confirming  actions  taken  during 
development  and  not  necessarily  a  result  of  a  lack  of  action  on  the  part  of  the  original  PM 
staff. 

The  next  chapter  provides  an  organized  analysis  of  the  data  presented  in  this 
chapter.  The  author  focuses  on  common  inhibitors,  enablers,  issues,  and  risks  associa  ted 
with  effective  reliability  management  while  discussing  mitigation  techniques. 
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V.  PROGRAM  RELIABILI TY  ANALYSIS  AND  LESS ONS 

LEARNED 


A.  INTRODUCTION 

This  chapter  provides  an  analysis  and  assessment  of  the  common  reliability 
management  issues  faced  by  Marine  Corps  Program  Management  Offices.  The  research 
results  focus  on  the  perspectives  and  opinions  of  acquisition  personnel,  as  attained  from 
the  survey  responses,  and  do  not  specifically  address  technology  driven  reliability 
problems.  The  analysis  follows  the  format  of  the  data  presented  in  the  previous  chapter, 
organized  around  the  five  reliability  management  themes,  developed  for  the  purpose  of 
this  thesis,  and  listed  below: 

•  Management  Approach  to  Reliability 

•  Determining  and  Documenting  Reliability  Requirements 

•  Contracting  and  Incentivizing  for  Reliability 

•  Reliability  Testing 

•  Comparing  and  Assessing  Required,  Estimated,  and  Achieved  Reliability 

While  the  research  is  limited  to  selected  legacy  principle  end  items,  it  is  logical 

that  many  of  the  challenges,  issues,  and  potential  solutions  correlate  to  other  end  items  in 
the  Marine  Coips  acquisition  process  or  currently  in  operational  use. 

B.  ANALYSIS  OF  RELIABILITY  MANAGEMENT  ISSUES 

1.  Management  Approach  to  Reliability 

a.  Factors  Contributing  to  Reliability  Performance  Problems 
Weapon  systems  often  fall  short  of  the  desired  level  of  reliability  planners 
and  developers  originally  planned  to  achieve.  While  increased  system  complexity  and 
harsh  environmental  and  operational  conditions  likely  contribute  to  the  challenges 
associated  with  achieving  reliability  requirements,  the  researcher  observes  that  a  large 
portion  of  performance  problems  can  be  directly  linked  to  management  of  reliability 
factors.  The  following  survey  responses  cite  the  top  five  reliability  issues  in  order  merit 
ranking  as  viewed  by  acquisition  professionals  participating  in  this  research: 

•  Wrong  Metrics  -  Traditional  T&E  RAM  metrics  are  not  supported  by 
USMC  maintenance  data  sources,  and  programs  are  unable  to  make  a 
valid  comparison  between  RAM  requirements  and  estimates  with  achieved 
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field  data.  How  can  a  program  manage  something  that  is  not  adequately 
measured? 

•  Schedule  Goals  Outweigh  Reliability  -  too  much  pressure  to  field  systems 
rapidly 

•  Need  more  qualified  reliability  personnel  in  Progr  am  Offices 

•  Unrealistic  Requirements  -  unrealistic  reliability  requirements  with 
inadequate  rationale;  there  seems  to  be  a  disconnect  between  user, 
materiel  developer,  and  combat  developer  thoughts  on  realistic 
requirements 

•  Poor  Reliability  Planning  and  Growth  Planning  -  reliability  growth  is  not 
properly  utilized  as  a  tool  to  reduce  reliability  related  issues  upfront 

b.  Reliability  Roles  and  Responsibilities 

Among  those  systems  examined,  there  was  no  consistent  managerial 
approach  concerning  who  was  delegated  responsibility  for  reliability  activities.  However, 
all  but  one  program  cited  at  least  one  individual  who  was  primarily  responsible  for 
reliability  as  survey  respondents  identified  Logistics  Team  Leaders,  Project  Team 
Leaders,  Supportability  Team  Leaders,  or  Reliability  IPTs. 

As  one  former  PM  noted,  PMs  often  manage  by  exception  and  without  a 
specific  problem  or  issue,  reliability  and  the  other  engineering  disciplines  are  managed 
through  empowerment  of  technical  experts.  In  this  sense,  it  is  interpreted  that  PMs 
depend  upon  outside  reliability  competency  for  matrix  support.  The  reliability  experts 
typically  provide  input  on  the  specific  reliability  activities  and  where  they  should  be 
implemented.  How  such  an  approach  is  incorporated  into  a  program  is  dependent  upon 
available  funding,  as  well  as  the  PM’s  judgment  based  on  cost,  schedule,  performance, 
and  supportability  considerations. 

Survey  respondents  identified  the  need  for  more  qualified  reliability 
personnel  in  program  offices  as  the  third  top  inhibitor  to  effective  reliability  management. 
By  assigning  a  permanent  reliability  engineer  or  staff,  just  as  the  AAAV  program  has 
done,  adequate  technical  expertise  is  available  to  the  Program  Manager.  This  would 
allow  logistics  engineers  to  partner  with  their  counterparts  in  reliability  engineering  to 
collectively  define  and  allocate  reliability  requirements  affecting  logistics.  The  duo  must 
defend  the  logistics  support  concepts  and  supportability  design  requirements  that  they 
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propose,  not  only  from  the  logistics  community’s  perspective,  but  also  from  the 
engineering  point  of  view. 

The  author  believes  that  assigning  an  individual  or  team,  such  as  a 
formally  chartered  reliability  IPT,  as  the  central  authority  on  reliability  activities  ensures 
there  is  a  reliability  advocate  that  can  defend  related  issues  and  identify  concerns  during 
the  numerous  trade-off  analyses  and  discussions. 

c.  Documenting  a  Program ’s  Reliability  Approach 

The  fifth  ranked  inhibitor  to  effective  reliability  management,  as  der  ived 
from  the  survey,  was  poor  reliability  planning  and  growth  planning.  Consequently,  half 
of  the  programs  reviewed  indicated  that  there  was  no  formal  reliability  management  plan 
while  the  other  half  simply  relied  on  the  contract  SOW  or  TEMP  to  address  the 
methodologies  and  plans  for  measuring  and  achieving  reliability.  Again,  the  exception 
was  the  AAAV  program  office,  which  uses  a  FRACAS  database  to  formally  document  its 
reliability  program  plan. 

In  order  to  provide  visibility  into  the  management  and  activities  of  those 
parties  responsible  (Government  and  contractor)  for  the  reliability  progress  within  a 
program,  there  should  be  definitive  documentation  on  all  reliability  activities,  functions, 
processes,  test  strategies,  measurement/metrics,  data  collection,  resources  and  timelines 
required  to  ensure  reliability  system  maturation.  Specifically,  Reliability  Program  Plans 
(RPPs)  can  serve  as  a  comprehensive  document  detailing  all  of  the  actions,  functions, 
resources  and  timelines  related  to  reliability.  Such  plans  would  especially  prove 
invaluable  if  an  initiative  is  implemented  that  would  make  predicted  and  demonstrated 
reliability  a  mandatory  component  of  the  acquisition  world  at  each  phase  of  the 
acquisition  cycle. 

d.  Tools  Used  in  Evaluating  Poten  tial  Failures 

Proactive  reliability  management  early  in  the  lifecycle  of  a  system  is 
typically  more  cost  effective  than  coping  with  schedule  delays  and  unanticipated  costs  of 
failing  a  test  later,  forcing  costly  redesign  and  additional  testing  to  demonstrate  that  the 
problem  is  corrected.  There  are  numerous  test  and  design  tools  available  to  program 
offices  and  contractors  that  help  to  ensure  reliability  is  “designed  in”  early  in  the 

program,  but  it  is  up  to  the  program  management  to  ensure  such  opportunit  ies  are 
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exploited.  The  risk  mitigation  techniques  consist  of  testing,  engineering,  and  other 
technical  methods  used  to  determine  and  evaluate  potential  system  failures  and  their 
causes.  However,  the  extent  to  which  the  legacy  programs  failed  to  utilize  t  he  reliability 
risk  mitigation  activities  was  surprising.  Aside  from  developmental  and  operational 
testing,  programs  generally  indicated  use  of  only  one  of  the  following  techniques: 
Environmental  Stress  Screening  (ESS),  reliability  modeling,  FMECA,  Reliability 
Development/Growth  Test  (RD/GT),  or  FRACAS.  No  (legacy)  programs  were  aware  of 
the  use  of  reliability  allocation,  fault  tree  analysis,  probabilistic  failure  assessment, 
reliability  qualification  test,  PFMEA,  Weibull  analysis,  physics  of  failure,  or  a  parts 
control  program.  The  AAAV  program,  currently  under  development,  seems  to  be  setting 
precedence  for  future  weapons  systems  in  terms  of  reliability  management,  as  it  was  the 
only  system  examined  which  utilized  all  of  the  identified  risk  mitigation  techniques 
identified  by  the  survey. 

Many  program  representatives  were  either  not  aware  of  the  specific 
techniques  utilized  to  ensure  reliability  was  “built  -in,”  or  the  original  staffs  did  not  use 
the  available  tools.  However,  there  was  a  common  consensus  to  test  early  and  often,  and 
use  knowledge  of  reliability  growth  to  implement  corrective  action.  All  PMOs  reported 
using  some  form  of  failure  analysis  as  an  integral  part  of  the  design  process,  and  there 
was  a  consensus  that  the  use  of  such  tools,  which  incorporate  reliability  prediction  and 
achievement  into  system  design,  was  beneficial. 

In  most  cases,  the  survey  respondents  were  not  the  original  PMO  staff,  and 
the  respondents  may  have  been  aware  of  which  techniques  were  utilized  only  by 
reviewing  any  existing  documentation  that  was  retained  before  their  arrival.  Often, 
documentation  from  the  original  staff  or  the  contractor  was  no  longer  available,  and  thus, 
the  assumption  that  the  legacy  systems  did  not  take  advantage  of  the  reliability  analysis 
tools  may  be  invalid.  In  reality,  many  of  the  programs  may  have  utilized  the  tools  more 
than  indicated  in  the  survey,  and  the  respondents  were  not  aware  of  the  previous  staffs’  or 
contractors’  actions. 

e.  Existing  Policy,  Guidance,  and  Regulations  on  Reliability 

The  acquisition  community  either  has  little  guidance  or  is  not  aware  of 

guidance  concerning  reliability  management.  Survey  responses  indicate  that  the  little 
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existing  guidance  is  very  broadly  scoped,  providing  minimal  detail  as  to  specific 
reliability  actions  to  be  taken  in  the  acquisition  process.  For  example,  the  DoD  5000.2 -R 
states  that  the  “PM  shall  establish  RAM  activities  early  in  the  acquisition  cycle.” 
Additionally,  the  amount  of  mandatory  guidance  is  minimal  and  has  further  decreased  in 
recent  years,  due  to  acquisition  reform  initiatives,  and  the  applicability  of  discretionary 
guidance  seems  to  be  somewhat  confusing  to  the  acquisition  community.  Ironically,  the 
DoD  5000  series  was  cancelled,  while  the  author  was  collecting  research  for  this  thesis, 
because  its  guidance  was  deemed  too  restrictive  in  nature. 

The  acquisition  workforce,  responsible  for  reliability  activities,  appears  to 
need  enforceable  and  precisely  delineated  criteria,  standards,  and  procedures  to  guide  it  in 
the  effective  management  of  reliability. 

2.  Determining  and  Documenting  Reliability  Requirements 
a.  Influencing  Realistic  Reliability  Requirements 
According  to  the  survey  responses,  which  indicate  the  materiel  developer 
was  able  to  provide  input  for  establishing  reliability  requirements  in  nearly  all  programs, 
it  appears  that  programs  actively  participate  with  the  Combat  Developer  to  determine  the 
requirements  relating  to  reliability.  Ironically,  the  fourth  top  rated  factor  contributing  to 
reliability  management  problems,  as  derived  from  the  first  survey  question,  was  that 
PMOs  were  faced  with  “unrealistic  reliability  requirements  with  inadequate  rationale.” 
This  indicates  that  there  is  a  disconnect  between  the  user,  materiel  developer,  and  combat 
developer  with  regard  to  the  perspectives  of  realistic  requirements. 

The  respondents  indicated  the  opportunity  is  available  to  influence 
reliability  requirements  generation.  It  is  essential  that  logistics  and  reliability  expertise 
be  involved  in  requirements  generation,  on  behalf  of  the  PM,  in  this  early  stage  of 
program  development.  The  extent  to  which  the  level  of  influence  is  effective  is  largely 
dependent  upon  their  level  of  expertise  and  involvement. 

A  review  of  the  terms  in  which  the  reliability  requirement  is  identified 
varies  from  program  to  program,  indicating  that  there  is  not  a  standard  operational 
terminology  in  which  reliability  must  be  expressed.  While  this  likely  allows  for 
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flexibility,  there  must  be  an  agreement  and  understanding  between  the  Government  and 
contractor  of  those  terms,  as  examined  in  upcoming  sections. 

b.  Reliability  as  a  KPP  in  the  ORD 

While  reliability  requirements  were  not  identified  as  KPPs,  programs 
agreed  that  reliability  was  an  important  priority  that  received  attention  in  the  ORD. 
According  to  DoD  5000. 2-R,  reliability  requirements  are  to  address  “mission  reliability” 
and  “logistics  reliability,”  implying  that  ORD  requirements  should  focus  on  measures 
related  to  mission  completion,  such  as  operational  availability.  Appropriately,  all  of  t  he 
systems  examined  had  specific  MTBOMF  or  MMBF  requirements. 

The  AAAV,  which  is  still  under  development,  is  the  only  program  that 
indicated  the  use  of  reliability  as  a  KPP.  In  fact,  the  AAAV  has  a  very  specific 
MTBOMF  threshold  as  a  KPP  for  the  Milestone  C  decision.  To  ensure  that  the 
requirement  is  in  an  objective  and  quantifiable  term  that  the  contractor  and  the 
Government  can  agree  upon,  according  to  the  reliability  survey  responses,  the  AAAV 
contractor  was  “given  the  Failure  Definition  and  Scoring  Criteria  which  was  the  basis  of 
determining  whether  a  failure  was  an  operational  mission  failure.” 

One  assumption  for  the  traditional  systems  not  designating  a  definitive 
reliability  KPP  is  that  doing  so  would  have  compromised  the  flexibility  and  trade-off 
range  available  to  PMs  in  a  time  where  cost,  schedule,  and  performance  were  the  priority. 
However,  it  is  generally  agreed,  in  theory,  that  reliability  and  maintainability,  along  with 
performance,  should  act  as  equal  partners  in  today’s  requirements  generation  process. 
This  is  logical  as  reliability  and  maintainability  contribute  to  combat  power  generation 
and  are  not  severable  from  system  performance.  If  reliability  is  not  a  KPP  in  the  ORD,  it 
simply  gets  pushed  aside  due  to  other  requirements  precedence. 

c.  Reliability  as  a  Source  Selection  Factor 

With  the  exception  of  the  AAAV,  the  program  respondents  replied  that 
either  reliability  was  not  a  factor  in  source  selection  or  they  were  not  certain  if  reliability 
was  a  factor  in  source  selection  due  to  the  time  that  had  passed  since  the  program  was 
originally  contracted  and  the  lack  of  documentation  in  the  PM  offices.  While  reliability 
was  not  a  source  selection  factor,  some  respondents  gave  the  impression  that, 

“Reliability,  with  its  impact  on  O&S  cos  ts,  should  receive  critical  attention  in  the  market 
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investigation,  solicitation,  and  source  selection  process.”  Reliability  is  a  logical  key 
source  selection  criterion  to  contribute  to  future  equipment  readiness  improvements  and 
to  ensure  adequate  logistics  weight  in  source  selection. 

3.  Contracting  for  Reliability 

The  concept  of  reliability  is  often  utilized  without  precise  definition,  while  the 
terminology  is  non-standard  throughout  the  logistics  community  and  tends  to  depend  on 
the  system  being  developed.  However,  while  creating  DoD  requirements  documentation, 
contract  specifications,  and  test  documentation,  it  is  very  important  that  all  main  concepts 
are  addressed  in  an  unambiguous  way  so  that  all  parties  involved  (to  include  the  user, 
combat  developer,  materiel  developer,  PM,  contractor,  and  tester)  understand  the  terms. 

A  common  problem  occurs  during  the  translation  of  operational  reliability 
requirements  into  contractual  reliability  requirements,  as  they  typically  are  expressed  in 
different  terms.  Operational  reliability  parameters,  as  derived  by  the  user  and  Combat 
Developer,  are  often  in  terms  of  operational  availability  or  mission  duration  needs.  The 
Materiel  Developer  takes  these  parameters  and  allocates  them  to  technical  reliabilities  of 
the  system  in  the  terms  of  MTBM,  MTBOMF,  or  MTBF,  traditionally  the  common 
parameters  of  reliability  used  for  contractual  purposes.  The  focus  for  the  PM  is  to  ensure 
the  contractual  reliability  of  the  system,  as  measured  in  controlled  test  conditions, 
supports  the  dynamic  and  unpredictable  environment  in  which  operational  availability  is 
measured.  The  challenge  for  PMs  remains  in  creating  a  definitive  correlation  between 
operational  and  contractual  reliability,  one  that  positively  indicates  anticipated  system 
reliability  performance. 

In  order  to  achieve  the  reliability  requirement  specified  in  the  ORD,  additional 
levels  of  increased  reliability  may  be  added  to  the  contract,  just  as  the  AAAV  has  done. 
This  helps  to  account  for  the  environmental  and  operational  differences  imposed  during 
fleet  operations.  However,  roughly  two-thirds  of  the  participating  survey  respondents 
indicated  that  the  ORD  paragraphs  relating  to  reliability  were  simply  restated  in  the 
Statement  of  Work  or  performance  specifications,  indicating  that  the  contract 
requirement  was  very  similar  to  the  ORD  requirement  and  open  to  interpretation. 
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The  contractor  and  designer  must  consider  the  risks  of  field  maintenance  and 
minimize  the  characteristics  of  the  design  that  are  susceptible  to  operationally  induced 
reliability  deterioration.  Likewise,  contractor  reliability  predictions  should  be  based  on 
realistic  operational  projections  for  degradation.  The  operation  and  maintenance  of 
equipment  in  the  field  can  induce  these  effects  by  stressing  sys  terns  beyond  predicted 
levels.  Operational  contributors  to  such  overstresses  include  neglect,  unfamiliarity, 
carelessness,  and  mission  constraints.  Maintenance  actions  can  also  induce  defects  in 
otherwise  satisfactory  assemblies;  foreign  objects  introduced,  fasteners  improperly 
engaged,  contaminants  introduced,  improper  part  replacement,  improper  lubricants,  etc. 
The  contract  provisions  should  attempt  to  account  for  all  elements  contributing  to  the 
combined  failure  rate  and  provide  the  Government  with  a  confidence  interval  for  a 
predetermined  readiness  performance  in  the  form  of  operational  availability.  Ultimately, 
performance  specifications  should  include  an  inherent  reliability  goal,  and  when  such 
goals  are  not  achieved,  it  should  be  considered  a  latent  defect. 

4.  Reliability  Testing 

a.  Resource  Constraints 

A  common  perspective  expressed  by  the  program  offices  is  a  lack  of  time 
and  money  to  conduct  adequate  levels  of  reliability  testing,  which  are  needed  to  achieve  a 
substantial  confidence  level  of  the  system  reliability.  In  fact,  data  from  the  first  survey 
question  indicated  that  there  is  too  much  pressure  to  field  systems  quickly  and  too  much 
pressure  to  field  systems  cheaply,  which  were,  respectively,  the  second  and  ninth  ranked 
inhibitors  to  effective  reliability  management.  However,  this  is  not  surprising  in  the 
acquisition  world  where  program  offices  continuously  conduct  trade-off  analyses  while 
competing  for  constrained  resources  in  a  politically  affected  environment. 

Funding  and  time  constraints  create  trade-offs  between  reliability 
improvements  and  performance  improvements.  In  reality,  system  performance 
improvements  are  often  made  at  the  expense  of  potential  reliability  enhancements.  Such 
decreased  emphasis  on  reliability  resources  typically  increase  life  cycle  costs  as  the 
acquisition  community  and  those  guiding  it  must  acknowledge  that  funding  not  spent  on 
reliability  optimization  during  development  will  be  spent  multiple  times  over  during 
operations  and  support. 
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Trading  off  reliability  for  another  performance  enhancement  is  not  only 
costly  in  terms  of  Total  Ownership  Costs,  but  may  actually  make  the  system  less  combat 
effective  as  maintenance  workload  increases,  adversely  affecting  system  availability. 

b.  Testing  to  Determine  Reliability  Performance 

A  major  focus  for  the  PM  is  to  ensure  the  contractual  reliability  of  the 
system,  as  measured  in  controlled  tests,  is  adequate  for  the  dynamic  and  unpredictable 
operational  environment  in  which  the  system  is  intended  to  operate.  Without  adequate 
Government  testing,  which  is  to  provide  a  substantial  confidence  level  of  system 
reliability,  DoD  is  often  forced  to  utilize  erroneous  contractor  or  manufacturer  estimates 
of  reliability  performance  as  a  basis  for  logistics  supportability  decisions.  Under-  or 
over-estimating  reliability  will  cause  limited  funds  to  be  allocated  unwisely.  For 
example,  inaccurate  reliability  estimates  can  potentially  have  devastating  effects  in  terms 
of  spare  parts  availability,  the  amount  and  level  of  mechanic  training,  repair  facilities 
infrastructure,  system  operational  availability,  and  increased  life  cycle  costs.  The 
contractor  or  manufacturer  claims  of  reliability  must  be  proven  through  independent 
testing,  and  claims  should  be  considered  unsubstantiated  until  tested.  In  other  words, 
DoD  must  adopt  the  null  hypothesis  that  the  reliability  is  not  what  the  contractor  claims, 
but  rather,  what  the  contractor  proves.  To  do  so,  the  Government  must  validate  the 
contractor’s  estimates.  Once  the  contractor  submits  their  RAM  estimate,  it  becomes  the 
Government’s  estimate  only  if  they  accept  it.  Therefore,  it  is  up  to  DoD  to  become 
involved  in  this  process  and  to  conduct  independent  testing  as  necessary  to  verify  such 
estimates. 

c.  Political  and  Cultural  Barriers  Affecting  Testing 

Program  Managers  are  evaluated  on  procurement  cost,  schedule,  and 
system  performance,  in  accordance  with  Defense  Acquisition  Executive  Summary, 
versus  supportability,  in  the  form  of  a  target  operational  availability  or  life  cycle  cost. 
Consequently,  PMs  have  traditionally  had  the  incentive  to  reduce  up-front  procurement 
costs  and  field  systems  rapidly,  often  accomplished  through  reduced  research, 
development,  test,  and  evaluation.  The  effects  of  such  incentives  directly  contribute  to 
reduced  reliability  performance,  decreased  readiness,  and  increased  life  cycle  costs.  If 
the  PMs,  the  prime  contractor,  and  all  members  of  the  IPTs  were  evaluated  on  readiness 


99 


performance,  and  incentives  were  in  place  to  reward  for  reliable  and  supportable  systems, 
the  effect  would  likely  be  an  increased  focus  on  RDT&E  activities  associated  with 
reliability  achievement,  which  would  ultimately  decrease  life  cycle  costs  while  increasing 
operational  availability. 

Commercial  firms  have  tended  to  adopt  more  successful  test  evaluations 
because,  as  a  whole,  they  have  an  appreciation  for  “why”  testing  is  conducted  vice  “how” 
testing  is  conducted.  In  general,  the  culture  of  the  commercial  industry  views  testing  as  a 
method  of  discovering  problems  early  which  results  in  less  expenses  later.  Corporate 
support  for  new  product  development  defuses  test  results  as  a  threat  to  program  support. 
Conversely,  it  is  difficult  for  weapon  system  programs  to  compete  for  approval  unless 
they  offer  significantly  better  performance  over  competing  systems  while  conforming  to 
funding  and  schedule  constraints.  In  this  sense,  test  results  tend  to  become  scorecards 
that  indicate  whether  a  program  is  ready  to  proceed  or  to  receive  the  next  increment  of 
funding,  an  activity  that  is  seemingly  intended  more  for  the  decision  makers  above  the 
program.  Thus,  program  managers  have  incentives  to  postpone  difficult  tests  while 
minimizing  open  communication  about  test  results  (GAO,  “A  More  Constructive  Test 
Approach  .  .  .,  p.  8). 

An  initiative  to  make  reliability  estimates  and  reliability  achievement 
demonstration  a  mandatory  part  of  each  phase  of  the  acquisition  cycle  would  alleviate  the 
incentives  for  program  managers  to  postpone  difficult  testing.  If  implemented,  the 
initiative  would  help  ensure  the  attainment  of  increasing  levels  of  system  reliability  as  the 
program  matured,  by  requiring  demonstrated  levels  of  reliability  before  major 
programmatic  approvals  and  milestones. 

5.  Comparing  and  Assessing  Required,  Estimated,  and  Achieved 

Reliability 

It  is  important  to  have  a  systematic  process  in  place  for  collecting  and  comparing 
reliability  data  for  several  reasons.  The  Marine  Corps  must  be  able  to  calculate  and 
compare  the  reliability  that  is  being  achieved  in  the  field  during  post -production  with  the 
required  and  estimated  reliability  in  order  to  determine  contactor  compliance, 
successfully  hold  contractors  to  their  estimates,  and  determine  if  the  user  reliability 
requirement  is  being  met. 
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The  basic  policy  of  DoD  is  to  hold  contractors  responsible  for  quality  of  the 
products  through  various  types  of  quality  assurance  programs.  This  obviously  requires  a 
plan  and  action,  which  must  be  based  on  the  quality  requirements  outlined  in  the  ORD. 
To  do  so,  it  is  recommended  that  a  program  use  the  reliability  requirements  stated  in 
operational  requirements,  or  those  resulting  from  trade-off  analysis,  as  a  baseline  for 
reliability  assessment  to  be  compared  with  actual  achieved  field  reliability.  However,  the 
difficulty  remains  in  collecting,  interpreting,  and  comparing  operational  (achieved) 
reliability  with  contractual  reliability  measurements.  Aside  from  the  essential  collection 
of  achieved  field  data,  original  contractor  estimates  and  ORD  requirements  must  be 
retained  for  comparison,  and  sustainment  organizations  mus  t  have  the  resources  to 
enforce  contractual  terms. 

It  is  well  known  that  “you  can’t  manage  what  you  don’t  measure,”  and  in  general, 
survey  responses  indicate  that  there  is  a  lack  of  a  systematic  process  for  collecting 
reliability  trend  data  beyond  readiness  ratings  for  Marine  Corps  ground  equipment. 
Furthermore,  what  data  that  does  exist  is  suspect  to  error  and  corruption  as  a  result  of  the 
current  maintenance  management  automated  information  systems.  There  was  also  a 
consensus  that  traditional  test  and  evaluation  RAM  metrics  are  not  supported  by 
maintenance  management  data  systems.  This  was  voted  as  the  top  rated  inhibitor  to 
effective  reliability  management,  according  to  the  responses  from  the  first  question  of  the 
survey.  With  these  factors  in  mind,  it  is  seemingly  impossible  to  compare  contractual 
reliability  and/or  test  estimates  in  the  form  of  MTBF  with  corrupted  operational 
reliability  data  in  the  form  of  MTBM,  as  attained  from  the  Marine  Corps  maintenance 
information  systems. 

a.  Maintaining  Reliability  Requirements  and  Reliability  Estimates 

Documentation  of  contractor  estimates  is  not  typically  retained  while 
ORD  reliability  requirements  are  often  difficult  to  locate.  None  of  the  participating 
programs  were  able  to  identify  the  level  to  which  the  contractors’  estimates  were 
demonstrated  (during  testing  phases  or  sustainment)  due  to  the  fact  that  contractor 
estimates  were  not  retained.  In  some  cases,  the  ORD  could  not  be  found,  meaning  the 
original  reliability  requirement  was  not  retained.  Thes  e  facts  indicate  that  there  is  an 
apparent  traceability  issue  within  the  program  offices  of  legacy  systems. 
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b.  Calculating  Achieved  Field  Reliability 

Most  of  the  programs  surveyed  indicated  that  achieved  field  reliability 
data  compiled  during  the  sustainment  phases  of  the  respective  systems  is  suspect  to  error, 
making  the  comparison  of  such  data  with  the  contractor  estimates  and  original  ORD 
requirements  (when  available)  questionable. 

When  attempting  to  compare  reliability  requirements,  reliability  estim  ates, 
and  achieved  reliability,  there  are  numerous  data  deficiencies  that  the  Marine  Corps  has 
been  forced  to  overcome.  First,  the  calculation  of  MTBF  is  unfeasible  at  this  time 
utilizing  the  traditional  maintenance  data  available  from  current  maintenance 
management  systems.  Next,  it  is  arguable  whether  MTBM  is  a  feasible  surrogate  for 
MTBF,  due  to  the  inclusion  of  preventive  maintenance  actions  used  in  calculating  this 
measurement.  Even  so,  the  data  used  for  MTBM  calculation  is  often  skewed  for  various 
reasons,  discussed  in  following  sections.  Thus,  the  logistics  and  program  management 
communities  have  been  forced  to  use  the  “next  best  measurement,”  operational 
availability,  to  attempt  to  measure  system  reliability.  The  question  remains  whether 
operational  availability  can  be  used  or  translated  to  make  a  valid  comparison  with  the 
contractors’  original  reliability  estimates  in  terms  of  MTBF,  MTBOMF,  MTBM,  etc. 

Mean  Time  Between  Failure.  It  is  important  to  distinguish  why  MTBF 
needs  to  be  calculated  for  equipment.  First,  contractor  reliability  estimates  are  typically 
provided  in  the  form  of  MTBF,  as  derived  from  operational  requirements.  As  a  result, 
the  contractual  reliability,  expressed  in  inherent  values  and  used  to  define,  measure,  and 
evaluate  the  contractor’s  program,  is  also  typically  in  the  form  of  MTBF.  In  order  to 
successfully  hold  contractors  to  their  estimates,  the  Marine  Corps  must  be  able  to 
calculate  the  reliability  that  is  being  achieved  in  the  field  during  post-production. 
Second,  the  calculation  of  this  time  is  also  necessary  in  order  to  determine  whether  the 
mean  time  between  failures  is  increasing,  decreasing,  or  remaining  constant  with  age.  As 
equipment  ages,  its  MTBF  decreases  until  the  cost  of  keeping  that  item  operational  is 
more  than  the  cost  of  buying  a  new  item.  Estimates  of  when  maintenance  costs  will 
exceed  acquisition  costs  are  questionable  without  mean  time  between  failure  calculations 
(Enholm,  p.  1).  In  other  words,  MTBF  data  analysis  helps  to  determine  or  confirm  if 

equipment  is  in  the  “wear-out”  phase  of  its  life  cycle  and  at  the  end  of  its  economic 
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useful  life.  Next,  the  determination  of  MTBF  can  serve  to  indicate  whether  Depot  Level 
Maintenance  programs  are  providing  a  cost  effective  benefit  by  comparing  reliability 
metrics  prior  to,  and  after  depot  level  maintenance,  allowing  decision  makers  to  tradeoff 
expected  readiness  for  DLM  cost  avoidance.  Lastly,  MTBF  could  be  used  as  an  input  to 
determine  the  optimal  provisioning  of  spare  parts,  utilizing  commercial  Readiness  Based 
Sparing  (RBS)  packages  and  techniques. 

Operational  usage  data  is  required  to  calculate  MTBF.  However,  the 
current  Marine  Corps  maintenance  management  data  systems  are  not  capable  of  tracking 
operational  usage.  Additionally,  there  are  concerns  as  to  which  time  unit  of  measurement 
is  most  appropriate  for  calculating  MTBF.  While  MTBF  is  traditionally  calculated  as  a 
time  between  failure,  other  units  such  as  mean  mileage  between  failures,  mean  operations 
or  starts  between  failures,  or  mean  rounds  fired  between  failures  may  be  better  indicators 
of  failure  intervals.  It  may  be  possible  to  construct  a  sophisticated  algorithm  to  determine 
a  mean  mileage  (or  other  form  of  operation)  for  specific  fleet  equipment,  but  the  error 
rate  would  be  relatively  high  due  to  the  often  inaccurate  and  unusable  meter  readings  of 
USMC  field  equipment. 

For  systematic  failure  or  maintenance  degradation,  there  is  also  a 
requirement  that  the  age  of  the  system  be  established.  Particularly,  this  is  important  when 
attempting  to  establish  the  age  when  a  system  starts  spending  more  time  in  a  non-mission 
capable  status  as  it  gets  older  and  maintenance  requirements  start  adversely  affecting 
operations.  However,  there  is  not  a  central  repository  or  data  source  where  the  Marine 
Corps  collects  information  establishing  the  economic  useful  life  of  serialized  items. 
While  the  program  management  offices  sometimes  keep  a  logbook  detailing  when 
specific  serial  numbers  were  fielded,  the  information  is  not  always  readily  available 
(Enholm,  p.  2). 

MTBF  vs.  MTBM  vs.  A-,  The  Marine  Corps  is  forced  to  substitute  MTBF 
with  either  MTBM  or  A,,  due  to  lack  of  operational  usage  data  needed  to  calculate 
MTBF.  The  feasibility  of  this  substitution  is  questionable  due  to  the  inclusion  of  both 
preventive  and  corrective  maintenance  actions  in  the  calculation  of  MTBM. 
Additionally,  even  the  calculation  of  MTBM  is  often  suspect  to  error  due  to  various 
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factors  contributing  to  a  lack  of  quality  maintenance  data,  and  as  indicated  in  the  previous 
chapter  as  well  as  Appendix  C,  MTBM  can  only  intermittently  be  calculated  for  certain 
equipment. 

As  a  result,  “R-rating”  or  readiness  rating  is  computed  per  Marine  Corps 
Bulletin  (MCBul)  3000  as  a  substitute  to  both  MTBF  and  MTBM,  offering  what  is 
perhaps,  the  most  feasible  estimate  of  a  measurement  for  reliability  achievement.  R- 
rating  basically  provides  a  snapshot  of  operational  availability  (A0),  as  discussed  in 
Chapter  II.  Recalling  that, 

UPTIME 

A  _  uptime  _  MTBM  _  OT  +ST 

uptime  +  dowtime  MTBM  +MDT  OT  +ST  +ALDT  +CMT  +PMT 

UPTIME  DOWNTIME 

it  is  important  to  note  that  the  Administrative  Logistics  Delay  Time  (ALDT),  Corrective 
Maintenance  Time  (CMT),  and  Preventive  Maintenance  Time  (PMT)  are  included  in  the 
calculation  of  downtime  when  computing  operational  availability.  This  is  largely  due  to 
the  fact  that  such  variables  are  difficult  to  extract  and  distinguish  amongst  one  another  as 
a  result  of  weaknesses  with  the  current  maintenance  management  data  systems  and 
tracking  procedures.  Operational  availability  can  also  be  improved  by  maintaining  a 
large  stockage  of  frequently  used  spare  parts.  While  availability  is  improved,  the 
logistics  burden  is  significant. 

Factors  Contributing  to  Skewed  Data.  As  indicated  by  a  majority  of  the 
survey  respondents,  there  is  an  obvious  lack  of  quality  historical  maintenance  data, 
attributable  to  numerous  causes,  some  of  which  are  highlighted  in  this  section. 

When  attempting  to  compare  the  required,  estimated,  and  achieved 
reliability  of  weapon  systems  with  the  limited  data  available,  it  is  important  to  take  into 
account  that  the  respective  readiness  rates  most  often  include  all  inventory  within  the 
Marine  Corps.  This  may  actually  skew  the  snapshot  of  overall  achieved  reliability  for 
particular  systems.  In  other  words,  reliability  based  on  readiness  rates  may  appear  to  be 
higher  than  what  is  really  being  achieved  due  to  the  equipment  in  stores  and  on  Maritime 
Pre-positioning  Force  (MPF)  ships  positively  affecting  the  overall  readiness  rates  for  a 
specific  system.  The  stores  and  MPF  equipment  have  an  extremely  low  usage  rate  and 
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are  well  maintained.  An  additional  data  set  that  excludes  stores  and  MPF  equipment 
would  be  required  to  test  this  hypothesis. 

Depot  Level  Maintenance  Programs  are  conducted  on  systems  that  have 
shown  trends  of  decreasing  readiness  corresponding  with  increased  age,  that  have 
exceeded  or  approached  their  expected  useful  lives,  or  that  are  will  be  required  to  be 
operational  for  extended  periods  without  a  replacement  system.  DLM  programs  are 
specifically  designed  to  have  a  significant  improvement  on  the  reliability,  availabilit  y, 
and  maintainability  of  the  systems.  As  a  result,  there  is  likely  an  impact  on  the  achieved 
reliability  data  that  is  collected  for  comparison  to  required  and  estimated  reliability.  In 
order  to  accurately  account  for  the  effects  of  improved  reliability,  the  data  collected  on 
the  systems  must  be  separated  into  those  that  have  undergone  DLM  and  those  that  have 
not. 

Selective  interchange,  replacement  of  major  system  components,  and 
maintenance  actions,  in  general,  affect  the  overall  inherent  reliability  that  a  system  was 
“designed  to.”  As  such  actions  change  the  reliability  and  availability  of  an  end  item,  it 
becomes  virtually  impossible  to  compare  what  the  system  was  supposed  to  achieve, 
according  to  Contractor/Government  estimates,  and  what  it  is  actually  achieving  in  the 
field.  Additionally,  “operator-  and  maintainer- induced  failures”  result  in  a  reliability 
achievement  that  is  less  than  the  inherent  reliability  of  the  system,  but  these  incidents  are 
also  largely  unknown. 

Lastly,  Equipment  Repair  Orders  (EROs),  used  to  initiate  maintenance 
activities,  often  include  multiple  failures  on  a  single  document,  creating  inconsistencies 
in  the  calculation  of  MTBM.  Also,  the  source  of  the  data  maintenance  and  supply  data  is 
from  two  disparate  systems,  MIMMS  and  ATLASS,  respectively. 

c.  “ Reliability  Gap” 

The  absence  of  retained  data  within  the  acquisition  and  logistics 
communities,  combined  with  the  lack  of  survey  participation,  led  to  an  inconclusive 
hypothesis  whether  a  “reliability  gap”  actually  exists  between  the  original  reliability 
requirements,  Contractor/Government  estimates,  and  achieved  field  reliability.  However, 
the  survey  respondents’  acknowledgements  to  an  absence  and  inaccuracy  of  the  requested 
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reliability  data  supports  the  concept  that  there  is  an  issue  with  respect  to  calculating, 
retaining,  comparing,  and  assessing  reliability  achievement  and  trends  of  mature  systems 
as  well  as  original  contractor  reliability  estimates  and  ORD  requirements.  In  this  case, 
“no  data,  is  data.” 

C.  LESSONS  LEARNED 

Program  Offices  of  Marine  Corps  legacy  weapon  systems,  procured  decades  ago, 
had  not  always  taken  advantage  of  effective  reliability  management  opportunities.  The 
proper  level  of  managerial  attention,  in  the  realm  of  reliability,  was  not  given  to  the 
systems  primarily  due  to  the  lack  of  focus  on  RAM  metrics  and  LCC  concerns. 
However,  it  is  promising  to  see  the  increased  focus  and  attention  given  to  reliability 
performance  of  the  AAAV  program,  likely  representative  of  future  programs  to  be 
developed  within  Marine  Corps  acquisitions. 

Today,  it  is  well  known  that  there  are  many  opportunities  to  address  reliability 
within  weapon  systems  acquisitions  -  from  requirements  generation  and  contracting  to  the 
conceptualization,  design,  and  development  utilizing  the  systems  engineering  process  to 
demonstration  and  incremental  testing  throughout  development  to  the  operational 
monitoring  and  comparing  of  achieved  reliability  with  the  estimated  reliability  to  ensure 
attainment  of  reliability  as  planned.  However,  procedural,  managerial,  and  incentive 
pressures  still  force  program  offices  to  sacrifice  reliability  for  the  achievement  of  other 
goals  such  as  time,  money,  and  performance. 

The  concept  of  reliability  is  often  utilized  without  precise  definition,  while  the 
terminology  is  non-standard  throughout  the  logistics  community  and  tends  to  depend  on 
the  system  being  developed.  However,  while  creating  DoD  requirements  documentation, 
contract  specifications,  and  test  documentation,  it  is  very  important  that  all  main  concepts 
are  addressed  in  an  unambiguous  way  so  that  all  parties  involved  (to  include  the  user, 
combat  developer,  materiel  developer,  PM,  contractor,  and  tester)  understand  the  terms. 
A  common  problem  emerges  in  the  translation  of  operational  reliability  requirements  into 
contractual  reliability  requirements,  as  they  typically  are  expressed  in  different  terms. 
Operational  reliability  parameters,  as  described  by  the  user  and  Combat  Developer,  are 
often  in  terms  of  operational  availability  or  mission  duration  needs.  The  Materiel 

Developer  takes  these  parameters  and  allocates  them  to  technical  reliabilities  of  the 
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system  in  the  terms  of  MTBF,  traditionally  the  common  parameter  of  reliability  used  for 
contractual  purposes.  Then,  the  focus  for  the  PM  is  to  ensure  the  contractual  reliability  of 
the  system,  as  measured  in  controlled  test  conditions,  supports  the  dynamic  and 
unpredictable  environment  in  which  operational  availability  is  measured. 

The  essential  documentation  and  collection  of  achieved  field  reliability  data, 
original  contractor  estimates,  and  reliability  requirements  are  not  typically  retained  or  are 
difficult  to  locate,  making  comparison  of  such  measurements  impossible.  In  fact,  none  of 
the  participating  programs  were  able  to  identify  the  level  to  which  the  contractors’ 
estimates  were  achieved  during  operational  testing,  developmental  testing,  and  the 
sustainment  phase  due  to  the  fact  that  contractor  estimates  were  not  retained  by  any  of  the 
programs  examined.  In  some  cases,  the  ORD  could  not  be  found,  meaning  the 
Government’s  original  reliability  requirement  was  not  retained  either.  Additionally, 
many  of  the  survey  participants  noted  that  developmental  test  data  was  difficult  to  come 
by  while  operational  test  data  was  somewhat  more  readily  available  due  to  the  role  of 
independent  Government  testing  agencies,  such  as  MCOTEA.  Lastly,  a  difficulty 
remains  in  collecting  and  computing  accurate  operational  reliability  data  as  most 
programs  indicated  that  achieved  field  reliability  data  compiled  during  sustainment  of  the 
respective  systems  is  suspect,  making  the  comparison  of  such  data  with  the  original  ORD 
requirement  and  contractor  estimate  (when  available)  questionable. 

Prior  to  research  and  data  collection  attempts,  PMOs  seemed  to  be  the  likely  place 
to  find  original  documents  and  data  such  as  ORDs,  reliability  requirements,  contracts, 
contractor  reliability  estimates,  and  achieved  reliability  data.  Research  ascertained  that 
there  was  a  retention,  traceability,  and  computation  problems  with  original  documents 
and  reliability  data. 

As  demonstrated  in  the  MTBF/Maintenance  Study  (Appendix  C)  conducted  at 
MARCORMATCOM,  it  is  possible  to  calculate  a  sample  mean  or  median  time  between 
failure  (and  maintenance  combined)  for  some  Marine  Corps  equipment  using  advanced 
methodologies,  but  the  accuracy  of  the  calculations  is  suspect  due  to  weaknesses  in 
maintenance  management  data  systems.  Likely  due  to  the  difficulty,  inability,  and  errors 
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involved  with  such  attempts,  there  is  not  a  requirement  to  track  the  MTBM  or  MTBF  of 
systems  during  their  operational  life  cycles. 

The  maintenance  data  available  using  the  Marine  Corps’  current  maintenance 
management  data  systems  (MIMMS/ATLASS  II/SASSY)  does  not  provide  the  adequate 
information  necessary  to  calculate  MTBF  and  is  suspect  to  corruption  and  error. 
Specifically,  the  Marine  Corps  is  unable  to  calculate  failure  rates  because  there  is  not  a 
way  to  track  the  operational  usage  of  end -items.  Perhaps  a  feasible  solution  will  arise 
with  the  implementation  of  the  Global  Combat  Support  System -Marine  Corps  (GCSS- 
MC).  The  GCSS-MC  system  will  replace  our  current  legacy  maintenance  systems  and 
could  possibly  contain  serialized  records  for  primary  end  items,  permitting  the  tracking  of 
operational  usage,  which  is  necess  ary  in  the  calculation  of  MTBF.  Navy  and  Marine 
Corps  supply/maintenance  procedures  used  to  track  flight  hours  on  naval  aircraft  may 
offer  possible  solutions  to  tracing  operational  usage  of  USMC  ground  equipment. 

Managing  reliability  does  not  end  with  OT&E  and  fielding  of  the  system,  and 
instead,  reliability  must  be  continually  monitored  and  assessed  for  potential 
improvements  and  efficiencies  in  support  of  meeting  Marine  Corps  life  cycle  cost  and 
readiness  objectives.  In  fact,  once  a  system  is  fielded,  reliability  assessment  should 
become  a  permanent  part  of  sustainment  activities  conducted  by  Program  Management 
Offices  as  well  as  other  Life  Cycle  Management  organizations.  To  be  successful, 
reliability  growth  must  continue  during  the  customer-use  phase  by  coordinating  feedback 
from  the  warfighters  to  the  suppliers  and  by  supporting  necessary  corrective  actions. 

D.  CHAPTER  SUMMARY 

This  chapter  analyzed  common  PM  issues  and  challenges  involved  with  managing 
the  reliability  performance  of  Marine  Corps  grou  nd  equipment.  The  analysis  was  based 
on  program  data  and  results  from  a  reliability  management  survey  administered  to 
various  personnel  associated  with  the  acquisition  process.  The  chapter  was  formatted 
around  the  five  major  reliability  management  issues,  derived  for  the  purpose  of  this 
research.  The  final  chapter  will  provide  selected  conclusions  focused  on  the  research 
questions  while  relaying  some  recommendations  on  how  to  best  approach  reliability 
issues  from  a  management  standpoint. 
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VI.  CONCLUSION  S  AND  RECOMMENDATION  S 


A.  INTRODUCTION 

Research  conducted  in  support  of  this  thesis  evaluated  system  reliability 
management  in  the  acquisition  process  as  it  applied  to  selected  principle  end  items  within 
the  Marine  Corps  ground  equipment  inventory.  Common  inhibitors  and  enablers  of 
effective  reliability  management,  why  they  occur,  lessons  learned,  and  potential  methods 
for  mitigating  the  inherent  risks  are  collectively  summarized  and  analyzed  as  derived 
from  surveys,  interviews,  and  existing  maintenance  data.  The  research  ascertains  a 
variety  of  technical,  programmatic,  managerial,  incentive,  and  procedural  issues  that  the 
Marine  Corps  encounters  concerning  system  reliability  requirements  and  achievement. 
The  overall  intent  of  this  research  is  to  provide  P  rogram  Management  Offices,  acquisition 
organizations,  and  strategic  planners  insight  into  the  collective  experiences  and 
perspectives  of  acquisition  workforce  professionals  who  are  familiar  with  issues  relating 
to  reliability  management.  The  results  of  the  thesis  serve  to  provide  insight  into  the 
improved  sustainability  of  future  systems  while  encouraging  the  development  of  a 
mechanism  that  enables  the  traceability  of  contractual  reliability  performance 
requirements  to  operational  reliability  requirements. 

In  this  final  chapter,  as  a  result  of  feedback  and  analysis  of  survey  responses, 
selected  conclusions  are  presented  that  focus  on  and  answer  the  research  questions,  while 
potential  recommendations  are  identified  on  how  to  best  approach  reliability  issues  from 
a  managerial  standpoint.  The  thesis  concludes  with  recommended  areas  for  further 
research. 

B.  CONCLUSIONS  AND  RECOMMENDATIONS 

Subtle  flaws  in  design  affect  system  reliability  and  can  have  multi-million  dollar 
effects  as  LCC  continue  to  escalate  throughout  the  life  of  the  system.  In  the  extreme, 
such  subtle  design  flaws  can  become  potentially  catastrophic.  The  DoD  needs  to  focus 
on  the  goal  of  providing  a  system  that  maximizes  its  operational  availability  within  the 
targeted  life-cycle  cost  of  the  program,  and  one  of  the  primary  methods  of  doing  so  is  to 
practice  effective  reliability  management. 


109 


1.  Documentation  and  Data  Retention  Required  for  Reliability 

Assessment 

Conclusion:  Prior  to  research  and  data  collection  attempts,  PMOs  appeared  to  be 
the  likely  place  to  find  original  documents  and  data  such  as  ORDs,  reliability 
requirements,  contracts,  contractor  reliability  estimates,  and  achieved  reliability  data. 
The  failure  to  retain  such  documentation  and  the  absence  of  reliability  data  within  the 
acquisition  and  logistics  communities,  combined  with  the  lack  of  survey  participation,  led 
to  an  inconclusive  hypothesis  whether  a  “reliability  gap”  actually  exists  between  the 
original  reliability  requirements,  Contractor/Government  estimates,  and  achieved  field 
reliability. 

However,  research  ascertained  that  there  was  a  retention  and  traceability  issue 
with  reliability  requirements,  reliability  estimates,  and  original  documents.  The  survey 
responses,  citing  an  absence  and  inaccuracy  of  the  requested  reliability  data,  indicate  that 
there  is  an  issue  with  respect  to  calculating,  comparing,  and  assessing  reliability 
achievement  and  trends  of  mature  systems  as  well  as  retaining  the  original  contractor 
reliability  estimates  and  ORD  requirements.  In  this  case,  the  author  believes  that  the  lack 
of  this  data  was  central  to  the  intent  of  the  thesis. 

Recommendation :  In  order  to  hold  contractors  to  their  original  reliability 
estimates,  it  is  recommended  that  a  baseline  for  reliability  assessment,  such  as  the 
reliability  requirements  stated  in  operational  requirements  and  performance 
specifications,  be  maintained  to  be  compared  with  achieved  field  reliability.  The  well  - 
meaning,  but  ineffectual  philosophy  often  applied  to  reliability  -  “we  will  do  the  best  we 
can”  must  be  replaced  with  a  contractual  obligation  in  the  form  of  measurable 
quantitative  system  reliability  requirements  and  appropriate  databases  of  fielded  system 
reliability  performance. 

2.  Reliability  Management  Control  Systems 

Conclusion :  The  Marine  Corps  acquisition  community  lacks  adequate 

management  control  systems  to  ensure  reliability  performance  meets  predetermined 
requirements  and  contractor  pre -fielding  estimates. 

Recommendation :  Mandate  that  required,  predicted,  and  demonstrated 

reliability  be  made  a  mandatory  component  of  each  phase  of  the  acquisition  cycle;  that  is, 
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require  that  realistic  reliability  predictions  and  demonstrations  of  achievement  be 
incorporated  into  each  Milestone  Decision  to  be  compared  with  the  original  requirements. 
This  would  allow  the  MDA  to  address  reliability  performance  progress  and  plans  for 
achieving  the  reliability  thresholds  of  a  system  at  every  major  review  of  a  program.  To 
do  so,  thresholds,  or  intermediate  benchmarks  representing  minimum  reliability 
achievement  levels,  should  be  established  at  various  points  of  the  program  as  a  risk 
management  technique.  A  breach  of  such  a  threshold  could  serve  to  indicate  that  the 
program  is  not  on  track  in  terms  of  reliability  requirements,  and  intervention  maybe 
required  to  correct  the  discrepancy,  if  necessary. 

The  Acquisition  Program  Baseline  should  continue  to  serve  as  a  “contract”  of 
sorts  between  the  PM  and  the  MDA.  Reliability  related  parameters  such  as  MTBF,  A0, 
and  MTBM  must  exist  for  each  program  either  in  the  Performance  or  Supportability 
sections  of  the  APB.  The  acquisition  program  baseline  status  of  each  program  must  be 
reviewed  at  regular  intervals  and  at  major  reviews. 

Additionally,  when  a  program  reaches  a  major  milestone  or  experiences  a 
significant  change  in  its  program  parameters,  the  outcome  is  documented  in  an  ADM. 
The  Acquisition  Decision  Memorandum  typically  includes  additional  directives  and 
statements  with  which  the  PM  must  comply,  and  thus,  it  needs  to  be  approached  as  an 
opportunity  for  the  MDA  to  encourage  the  achievement  or  improvement  of  reliability 
levels,  while  placing  entrance  criteria,  constraints,  or  follow-on  actions  related  to 
reliability  on  the  programs. 

3.  Reliability  Policy,  Regulations,  and  Guidance 

Conclusion :  Predominantly  as  a  result  of  acquisition  reform,  acquisition 

workforce  professionals  lack  adequate  guidance  and  requirements  in  the  form  of  policy, 
publications,  and  directives,  to  guide  them  in  the  conduct  of  reliability  activities. 
Additionally,  the  amount  of  mandatory  guidance  is  minimal  and  has  further  decreased  in 
recent  years  due  to  acquisition  reform  initiatives.  Consequently,  the  acquisition 
community  often  utilizes  an  abundance  of  cancelled  MIL-STDs  and  MIL-HDBKs  or 
vague  discretionary  guidance  to  guide  them  in  reliability  related  decision-making. 
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Recommendation :  DoD  should  couple  with  commercial  industry  prime 

contractors  to  develop  a  comprehensive  set  of  reliability  management  standards,  which 
consider  life  cycle  cost  and  sustainment  consequences  of  early  life  cycle  decisions. 

4.  Reliability  Roles 

Conclusion :  Throughout  the  program  offices  of  the  legacy  systems  examined, 
there  was  no  consistent  managerial  approach  to  reliability  responsibilities,  and  while  this 
may  allow  for  flexibility,  programs  sometimes  seemed  to  lack  a  clear  understanding  of 
who  is  responsible  for  reliability  activities  within  a  program. 

Recommendation :  There  should  be  a  formally  chartered  Reliability  IPT, 

responsible  for  ensuring  effective  communication  between  the  program,  us  er,  contractor, 
RAM  community,  engineers,  testers,  and  logisticians.  The  IPT  must  be  involved  from 
concept  exploration,  affecting  requirements  establishment,  analysis,  and  influence  over 
the  design  factors. 

5.  Reliability  Requirements  Generation 

Conclusion:  There  does  not  seem  to  be  a  consistent  process  for  establishing 
operational  reliability  requirements  performance  measures  during  attempts  to  “link 
reliability  performance  to  mission  or  supportability  measures”,  as  required  by  DoD 
5000. 2-R.  In  other  words,  the  terms  in  which  the  reliability  requirement  was  identified 
varied  from  program  to  program.  Additionally,  reliability  has  typically  not  been  a  Key 
Performance  Parameter  in  the  Operational  Requirements  Document  for  Marine  Corps 
legacy  systems,  and,  it  gets  pushed  aside  due  to  other  requirements  precedence  to  include 
cost,  schedule,  and  performance  objectives.  Lastly,  amongst  the  legacy  systems 
examined,  reliability  was  not  used  as  a  source  selection  factor. 

Recommendation :  It  is  recommended  that  standards  be  established  for  defining 
reliability  measures  in  the  ORD.  Reliability,  along  with  cost,  maintainability,  and 
performance,  should  be  considered  equally  in  the  requirements  generation  process,  a 
stage  at  which  the  Materiel  Developer  and  Combat  Developer  should  be  jointly  defining 
realistic,  achievable  reliability  requirements.  To  attain  desired  reliability  performance 
thresholds  and  goals,  it  is  recommended  that  robust  reliability  requirements  be  clearly 
defined  and  communicated  as  KPPs  in  the  ORD,  in  clear  operational  terms  by  the 

Combat  Developer.  Likewise,  reliability  should  appropriately  be  considered  as  a  source 
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selection  factor  during  solicitation.  The  reliability  objectives  must  then  be  translated  into 
quantifiable  and  verifiable  contractual  terms,  traceable  back  to  the  system  performance 
and  the  operational  requirements.  The  Materiel  Developer  must  adequately  translate  the 
system  operational  terms  into  viable  contractual  terms,  understood  by  all  parties  involved 
to  include  the  user,  the  program  office,  and  the  contractor  so  that  compliance  can  be 
adequately  monitored  and  enforced.  Ultimately,  it  is  recommended  that  Performance 
Specifications  include  an  inherent  reliability  goal,  and  when  such  goals  are  not  achieved, 
it  should  be  considered  a  latent  defect. 

6.  Reliability  Program  Plan 

Conclusion :  For  the  most  part,  Marine  Corps  legacy  programs  did  not  have 
structured  reliability  management  processes  in  place,  nor  did  they  have  corresponding 
overarching  documents  that  define  the  activities,  schedules,  resources,  and  reliability 
achievement  strategies  needed  to  provide  managerial  insight  into  the  programs’  reliability 
objectives.  Consequently,  Program  Management  Offices  have  not  traditionally  taken 
advantage  of  the  numerous  test  and  design  tools  available  to  them  and  contractors  that 
help  to  ensure  reliability  is  “designed  in”  early  in  the  program  by  determining  potential 
failures  and  the  causes  of  such  failures.  “Designing -in”  reliability  upfront  reduces  risk 
and  is  less  costly  than  finding  design  discrepancies  during  later  stages  of  testing, 
evaluation,  and  operational  use. 

Recommendation :  In  order  to  provide  visibility  into  the  management,  functions, 
and  responsibilities  of  those  parties  responsible  (Government  and  contractor)  for  the 
reliability  activities  within  a  program,  require  all  PMs  to  develop  a  Reliability  Program 
Plan.  This  should  be  a  mandatory  document  for  all  Milestone  Decision  Reviews, 
providing  definitive  documentation  on  all  reliability  activities,  functions,  processes,  test 
strategies,  measurements/metrics,  data  collection,  resources  and  timelines  required  to 
ensure  system  reliability  maturation. 

7.  Impact  of  Inaccurate  Reliability  Estimates:  Tying  Contractors  to 
their  Estimates 

Conclusion :  Foremost,  systems  that  fail  to  meet  reliability  goals  do  not  perform 
as  expected,  and  the  failure  to  meet  such  performance  expectations  proves  to  be  costly  in 
both  operational  and  financial  terms.  Logistical  support  decisions  are  based  upon 
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expected  system  reliability  as  determined  early  during  development.  Consequently, 
inaccurate  reliability  estimates  significantly  increase  life  cycle  costs  while  adversely 
affecting  the  quality  of  logistical  support  available  throughout  the  systems’  life  cycles.  In 
other  words,  when  achieved  field  reliability  is  significantly  less  than  what  was 
anticipated,  numerous  deficiencies  occur  in  the  attempt  to  support  maintenance 
requirements  to  include  inadequate  initial  provisioning  of  spare  parts,  insufficient  number 
and  level  of  training  for  mechanics,  deficient  facilities  and  test  equipment,  and  inadequate 
funding  plans  for  future  budgets. 

Recommendation :  Contractors  must  be  tied  to  LCC  through  their  reliability 
estimates.  Attempts  must  be  made  to  use  reliability  incentives  and  warranties,  and  the 
Marine  Corps  must  establish  a  mechanism  that  allows  for  the  traceability  of  contractual 
reliability  performance  requirements  to  operational  performance  requirements.  Likewise, 
the  DoD  must  adopt  the  null  hypothesis  that  the  reliability  is  not  what  the  contractor 
claims,  but  rather,  what  the  contractor  proves.  To  do  so,  the  Government  must  validate 
the  contractor’s  estimates.  Once  the  contractor  submits  their  RAM  estimate,  it  becomes 
the  Government’s  estimate  only  if  they  accept  it.  Therefore,  it’s  up  to  DoD  to  become 
involved  in  this  process  and  to  conduct  independent  testing  as  necessary  to  verify  such 
estimates. 

Another  alternative  is  to  utilize  Contractor  Logistical  Support  (CLS)  for  a 
specified  interim  period  to  ensure  contractors  are  tied  to  acc  urate  reliability  estimates 
prior  to  transitioning  the  logistical  support  role  to  the  military.  After  an  accurate 
operational  reliability  measurement  is  determined  through  field  operations,  the  Marine 
Corps  can  enforce  contract  provisions  and  optimally  plan  for  and  implement  logistics 
support  based  upon  actual  achieved  reliability.  Traditionally,  CLS  has  been  utilized  in 
cases  where  the  military  was  not  yet  in  a  position  to  provide  logistical  support  for  a 
system  that  needed  to  be  fielded  rapidly.  The  author  proposes  the  intentional  use  of 
planned  CLS  for  a  predetermined  period,  perhaps  two  to  three  years,  to  ensure  that  the 
system  reliability  is  what  the  contractor  had  claimed,  to  be  able  to  enforce  contract 
provisions,  and  to  plan  for  effective  system  supportability  to  achieve  desired  system 
performance. 
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8.  Reliability  Assessment  Metrics 

Conclusion :  Traditional  T&E  RAM  metrics  are  not  supported  by  current  Marine 
Corps  maintenance  data  sources,  and  thus,  it  is  difficult  to  make  a  valid  comparison 
between  reliability  requirements,  as  stated  in  the  ORD;  contractor  RAM  estimates;  and 
data  from  actual  achieved  reliability  of  fielded  systems.  Specifically,  USMC 
maintenance  management  systems  do  not  track  the  necessary  operational  usage  data 
required  to  accurately  compute  failure  rate  (in  the  form  of  MTBF),  and  instead,  the 
current  systems  enable  the  computation  of  maintenance  rate  (in  the  form  of  MTBM), 
which  includes  both  preventive  and  corrective  maintenance  as  well  as  other  skewing 
factors.  The  accuracy  of  the  MTBM  calculation  is  suspect  to  corruption  and  error  due  to 
weaknesses  in  maintenance  management  data  systems  and  inefficiencies  in  the 
administrative  maintenance  processes.  As  a  result,  it  is  extremely  difficult  to  accurately 
compare  contractual  reliability  and/or  test  estimates  in  the  form  of  MTBF  with  corrupted 
operational  reliability  data  in  the  form  of  MTBM,  as  attained  from  the  Marine  Corps 
maintenance  information  systems. 

Additionally,  the  concept  of  reliability  is  often  utilized  without  precis  e  definition, 
and  the  terminology  is  non-standard  throughout  the  logistics  community,  tending  to 
depend  on  the  system  being  developed.  However,  while  creating  DoD  requirements 
documentation,  contract  specifications,  and  test  documentation,  it  is  very  important  that 
all  performance  terms,  including  supportability  performance  terms,  are  addressed  in  an 
unambiguous  way  so  that  all  parties  involved  (to  include  the  user,  combat  developer, 
materiel  developer,  PM,  contractor,  and  tester)  understand  them.  A  commo  n  problem 
emerges  in  the  translation  of  operational  reliability  requirements  into  contractual 
reliability  requirements,  as  they  typically  are  expressed  in  different  terms.  The  focus  for 
the  PM  is  to  ensure  the  contractual  inherent  reliability  of  the  system,  as  measured  in 
controlled  test  conditions,  supports  the  dynamic  and  unpredictable  environment  in  which 
operational  availability  is  measured. 

Recommendation :  Foremost,  the  operational  and  contractual  reliability 

requirements  must  be  measurable,  verifiable,  and  most  importantly,  they  must  be 
comparable  in  an  objective  and  quantifiable  form  that  are  contractually  enforceable  and 

that  contractors  and  the  Government  can  easily  agree  upon. 
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The  author  recommends  that  a  feasible  solution  to  track  operational  usage, 
enabling  the  calculation  of  MTBF,  be  included  with  the  implementation  of  the  Global 
Combat  Support  System-Marine  Corps  (GCSS-MC).  The  GCSS-MC  system  will  replace 
current  USMC  legacy  maintenance  systems,  and  the  author  suggests  that  the  replacem  ent 
system  contains  serialized  records  for  primary  end  items,  permitting  the  tracking  of 
operational  usage,  which  is  necessary  in  the  calculation  of  MTBF.  Additionally,  weapon 
systems  should  be  designed  that,  through  modem  technology,  are  able  to  self  -monitor 
operational  usage  in  terms  of  hours,  starts,  rounds  fired,  miles,  or  other  applicable 
metrics.  Another  alternative  is  to  consider  Navy  and  Marine  Corps  supply/maintenance 
procedures  used  to  track  flight  hours  on  naval  aircraft  as  ways  to  offer  possible  solutions 
to  tracing  operational  usage  of  USMC  ground  equipment. 

9.  Cost  Metrics 

Conclusion :  Under  current  methods  of  utilizing  unit  production  cost  as  a  metric 
that  determines  the  success  of  a  program  for  the  MDA,  reliability  and  life  cycle  cost 
issues  are  often  ignored  to  produce  a  lower  unit  cost.  Likewise,  while  the  acquisition 
workforce  recognizes  the  significance  of  reliability,  it  typically  gets  pushed  aside  in  the 
short-term  crisis  management  environment  of  a  constrained  resources  acquisition 
community  where  PMs  are  evaluated  on  procurement  cost,  schedule,  and  performance. 

Recommendation :  Mandate  the  use  of  life  cycle  cost,  within  performance 
parameters,  as  the  basis  for  all  design  trade-offs.  However,  DoD  must  develop  a 
performance  measure  and  incentive  structure  that  recognizes  life  cycle  cost  equal  to,  or 
higher  than  the  current  acquisition  cost,  schedule,  and  performance  metrics. 

Additionally,  there  needs  to  be  a  greater  awareness  by  program  management 
offices  of  the  availability  and  capabilities  of  commercial  LCC  models.  Such  models 
should  be  utilized  to  assess  the  reliability,  LCC,  and  logistics  supportability  impacts  of 
various  equipment  configurations  and  other  design  and  supportability  issues.  The 
commercial  models  should  be  used  to  provide  design  and  support  tradeoff,  with 
sensitivity  and  comparative  analysis,  to  ensure  the  program  meets  adequate  reliability 
goals  within,  or  below  respective  LCC  budgets. 
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10.  Inherent  vs.  Achieved  Reliability:  Consideration  of  Reliability 
Degradation 

Conclusion :  There  will  likely  be  a  difference  between  inherent  (or  potential) 
reliability  and  achieved  reliability  as  demonstrated  in  the  field.  The  operation  and 
maintenance  of  equipment  in  the  field  often  induces  these  effects  by  overstressing 
systems  beyond  predicted  levels  as  a  result  of  neglect,  unfamiliarity,  carelessness,  and 
mission  constraints.  Additionally,  the  true  achieved  reliability  can  be  obscured  by 
scheduled  and  unscheduled  maintenance  actions  and  the  corresponding  administrative 
actions,  conducted  by  inadequately  trained  personnel,  mandated  by  incorrect  diagnosis, 
or  simply  poorly  managed. 

Recommendation :  While  a  major  effort  is  made  in  operations  to  reduce  the 
effects  of  reliability  degradation  caused  by  maintenance,  the  desig  ner  should  consider  the 
risks  of  field  maintenance  and  minimize  the  characteristics  of  the  design  that  are 
susceptible  to  operationally  induced  reliability  deterioration.  Every  effort  should  be 
made  by  both  the  Materiel  Developer  and  the  contractor  (throu  gh  contractual  language) 
to  minimize  potential  human  error.  This  may  be  accomplished  with  technologies  such  as 
bit/bite,  diagnostics,  prognostics,  and  autonomies.  Equally  important,  reliability 
predictions  should  be  made  on  realistic  operational  projections  for  degradation. 

Contractors  should  account  for  all  elements  contributing  to  the  combined  failure 
rate  and  provide  the  Government  with  a  confidence  interval  for  a  predetermined  readiness 
performance  in  the  form  of  operational  availability.  Such  concep  ts  remain  the  premise 
behind  the  idea  of  Performance  Based  Logistics,  which  is  a  strategic  approach  to  provide 
long-term  measurable  product  sustainment  to  the  warfighter.  A  realistic  reliability 
requirement  must  account  for  all  application  environments  and  the  time  proportions 
expected  in  each  phase  or  product  life  cycle  of  a  system  such  as  storage,  transportation, 
installation,  standby,  and  operation,  and  an  apportionment  of  the  requirement  across  the 
life  cycle  phases  must  account  for  deterioration  in  each  phase. 

11.  Continuous  Reliability  Assessment 

Conclusion :  Managing  reliability  cannot  end  with  OT&E  and  fielding  of  the 
system.  Typically,  legacy  weapons  systems  did  not  implement  formal  reliability  growth 
plans  to  ensure  established  reliability  maturity  levels  were  achieved  during  system 
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sustainment  periods.  Performance  thresholds  linked  to  system  sustainability  may  or  may 
not  have  been  met,  and  unfortunately,  the  USMC  does  not  know  which  is  the  case. 

Recommendation :  Reliability  must  be  continually  monitored  and  assessed  for 
potential  improvements  and  efficiencies  in  support  of  meeting  Marine  Corps  life  cycle 
cost  and  readiness  objectives.  Once  a  system  is  fielded,  formal  reliability  assessment 
should  become  a  permanent  part  of  sustainment  activities  conducted  by  Program 
Management  Offices  under  the  realm  of  the  life  cycle  Weapon  System  Manager  with 
assistance  from  the  Logistics  Management  Specialist.  To  be  successful,  reliability 
growth  must  continue  during  the  customer -use  phase  by  coordinating  feedback  from  the 
warfighters  to  the  suppliers  and  by  supporting  necessary  corrective  actions.  The  data 
required  for  this  effort  must  be  collected  in  a  method  that  is  transparent  to  the  operators 
and  maintainers. 

C.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

The  following  topics  could  have  significant  influence  in  the  area  of  reliability 
management  but  fall  beyond  the  scope  of  the  present  research,  and  thus,  they  are 
recommended  topics  for  additional  research: 

•  Conduct  a  quantitative  assessment  to  determine  the  level  of  confidence 
that  can  be  realized  when  substituting  MTBF  with  MTBM  or  A0 
calculations.  Such  a  study  could  help  to  more  accurately  determine  if 
these  measurements  are  feasible  surrogates  for  one  another. 

•  Conduct  a  similar  study  that  encompasses  a  larger  spectrum  of  programs, 
to  include  various  ACAT  levels  as  well  as  systems  in  various  acquisition 
phases. 

•  Conduct  a  similar  survey  conducted  from  the  perspective  of  the  contractor, 
who  is  expected  to  provide  systems  capable  of  achieving  a  predetermined 
required  level  of  reliability. 

•  Examine  the  feasibility  of  utilizing  Contractor  Logistical  Support  for  a 
specified  interim  period  to  ensure  contractors  are  tied  to  accurate 
reliability  estimates  prior  to  passing  the  logistical  support  role  onto  the 
military  once  actual  reliability  is  determined  through  field  operations. 

•  Analyze  the  applicability  and  feasibility  of  utilizing  reliability  warranties 
and  incentives  in  contracts  to  achieve  more  accurate  reliability  estimates 
from  contractors  to  which  they  can  be  help  accountable. 

•  Using  a  commonly  identified  commercial  life  cycle  cost  model,  conduct  a 
sensitivity  analysis  of  how  changes  in  reliability  affect  operations  and 
support  costs. 
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•  Conduct  a  study  to  determine  the  political  and  cultural  barriers  to 
increasing  RDT&E  investment,  a  funding  investment  that  although 
increases  procurement  costs,  may  lead  to  more  accurate  reliability 
estimates  and  ultimately  decrease  LCC. 

D.  THESIS  SUMMARY 

Weapon  system  reliability  demands  constant  managerial  attention  and 
implementation  of  effective  management  and  technical  strategies  that  balance  cost, 
schedule,  and  performance  with  reliability  during  systems’  entire  life  cycles  -  from 
conceptualization  and  development  to  fielding  and  operational  support  through  disposal. 
It  is  imperative  to  have  early  identification  of  upfront  cost-effective  opportunities  for 
achieving  the  required  reliability  while  obtaining  and  utilizing  accurate  reliability 
estimates  for  logistical  support  decisions.  Mitigation  of  associated  reliability  risks  during 
design,  manufacturing,  development,  testing,  and  post-production  operations  must  be 
accomplished  to  reduce  the  potential  for  unexpected  life  cycle  cost  inflation  and 
decreased  operational  availability.  Likewise,  programs  that  have  carefully  planned  and 
executed  reliability  management  techniques  will  be  more  combat  effective,  in  the  way  of 
decreased  life  cycle  costs  and  increased  operational  availability,  during  the  useful  life  of 
the  systems. 

The  bottom  line  remains  that  increased  reliability  of  weapon  systems  contributes 
directly  to  greater  combat  effectiveness.  To  fight  and  win  wars,  Marines  must  be 
equipped  with  systems  that  are  reliable.  Progress  must  continue  to  be  made  to  ensure  that 
inhibitors  to  effectively  provide  such  lethal  systems  to  the  fleet  are  recognized  and 
mitigated. 
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APPENDIX  A.  ACQUISITION  RELATED  ACRONYM  S 


ADM 

Acquisition  Decision  Memorandum 

ACAT 

Acquisition  Category 

Aa 

Achieved  Availability 

A 

Inherent  Availability 

ALDT 

Administrative  and  Logistics  Delay  Time 

Ao 

Operational  Availability 

AoA 

Analysis  of  Alternatives 

APB 

Acquisition  Program  Baseline 

ASN,  RDA 

Assistant  Secretary  of  the  Navy;  Research,  Development,  and 
Acquisition 

CAE 

Component  Acquisition  Executive 

CATV 

Cost  as  an  Independent  Variable 

CE 

Concept  Exploration 

CMC 

Commandant  of  the  Marine  Corps 

CMT 

Corrective  Maintenance  Time 

DAE 

Defense  Acquisition  Executive 

DAU 

Defense  Acquisition  University 

DFAR 

Defense  Federal  Acquisition  Regulation 

DLM 

Depot  Level  Maintenance 

DoD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 

DPG 

Defense  Planning  Guidance 

DSMC 

Defense  Systems  Management  College 

DT&E 

Developmental  Test  and  Evaluation 

DUSD 

Deputy  Under  Secretary  of  Defense 

ESS 

Environmental  Stress  Screening 

FAR 

Federal  Acquisition  Regulation 

FMECA 

Failure  Modes,  Effects  and  Criticality  Analysis 

FRACAS 

Failure  Reporting,  Analysis,  and  Corrective  Action 

GAO 

General  Accounting  Office 

ILS 

Integrated  Logistic  Support 

IOT&E 

Initial  Operational  Test  and  Evaluation 

IPPD 

Integrated  Product  and  Process  Development 

IPT 

Integrated  Product  Team 

IROAN 

Inspect  and  Repair  Only  As  Necessary 

121 


KPP 

Key  Performance  Parameter 

LCC 

Life  Cycle  Cost 

LRIP 

Low  Rate  Initial  Production 

LS 

Logistics  Supportability 

LT&E 

Logistics  Test  and  Evaluation 

M&S 

Modeling  and  Simulation 

MARCORSYSCOM 

Marine  Corps  Systems  Command 

MARCORLOGBASE 

Marine  Corps  Logistics  Base 

MATCOM 

Materiel  Command 

MCCDC 

Marine  Corps  Combat  Development  Command 

MCOTEA 

Marine  Corps  Operational  Test  and  Evaluation  Activity 

MDA 

Milestone  Decision  Authority 

MDAP 

Major  Defense  Acquisition  Program 

MIL-HDBK 

Military  Handbook 

MILSPEC 

Military  Specification 

MLDT 

Mean  Logistics  Delay  Time 

MNS 

Mission  Need  Statement 

MOE 

Measures  of  Effectiveness 

MS 

Milestone 

MIBF 

Mean  Time  Between  Failure 

MTBM 

Mean  Time  Between  Maintenance 

MTTR 

Mean  Time  to  Repair 

NAE 

Navy  Acquisition  Executive 

NMS 

National  Military  Strategy 

NSS 

National  Security  Strategy 

O&M 

Operations  and  Maintenance 

O&S 

Operations  and  Support 

ORD 

Operational  Requirements  Document 

OSD 

Office  of  the  Secretary  of  Defense 

OT 

Operating  Time 

OT&E 

Operational  Test  and  Evaluation 

PEO 

Program  Executive  Officer 

PM 

Program  Manager 

PMO 

Program  Management  Office 

PMT 

Preventive  Maintenance  Time 

PM/WSM 

Program  Manager/Weapon  System  Manager 

POM 

Program  Objectives  Memorandum 

RAM 

Reliability,  Availability,  and  Maintainability 

R&D 

Research  and  Development 

RDT&E 

Research,  Development,  Test,  &  Evaluation 
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RFP 

Request  for  Proposal 

RQT 

Reliability  Qualification  Test 

SA 

Supportability  Analysis 

SAE 

Senior  Acquisition  Executive 

SAMP 

Single  Acquisition  Management  Plan 

SCM 

Supply  Chain  Management 

SECDEF 

Secretary  of  Defense 

SEP 

Systems  Engineering  Process 

SLEP 

Service  Life  Extension  Program 

ST 

Standby  Time 

TAAF 

Test,  Analyze,  and  Fix 

TEMP 

Test  and  Evaluation  Master  Plan 

TOC 

Total  Ownership  Cost 

USD  (AT&L) 

Under  Secretary  of  Defense,  Acquisition,  Technology,  and 
Logistics 

USMC 

United  States  Marine  Corps 

WBS 

Work  Breakdown  Structure 

WIPT 

Working-Level  Integrated  Product  Team 
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APPENDIX  B.  PROGRAM  MANAGEMENT  RELIABIL ITY 

SURVEY 


New  P  age  2 


P-J0I  1  Of  7 


Naval  Postgraduate  Schoo 


anterry. 


C  i  I  I  I  o  r  r 


Reliability  Survey 

Instructions 

Himim  arftwoi  the  hi  I  lowing  quaUtkmc  NLT  U>  Oc tabor  n?  Your  llnie  arid  offoet  Is  g  ready  appreciated  and  vs«S 
hcpofUty  assist  m  Idcntitymu  common  potential  inMblUxs  of  rdiabruty  managamerK.  Results  from  this  sorter  wtU 
bm  prmxMntrd  in  <rg(r»V«M  form,  not  program  apmdfic. 

Hrrri"  fimw»r  all  niN^»lnpa  B  data  nr  r|nrrmii*nti*tin*i  I*  nnf  a  v-allatil**,  Himmi  not  exl*k.  nr  im*i  not  trrn  m«l|Nitnl, 
picas#*!  upeeffy. 

This  aui  «cf  is  being  conducted  to  support  research  as  port  of  a  Naval  Postgraduate  School  th&sie  on  reliability 
nwnirgeroent  choile«ges  wlthm  the  Marine  Corps  atgulsHion  pmcews  the  results  of  the  tt»e*e  a»e  site  ruled  to  drectly 
benefit  Program  Managers  L>»  dentil ylnQ  common  dhilXuts  to  reiabiity  management,  why  the)  occur,  lesson* 
irarnnd,  and  suggettlonfi  Vir  reducing  tb"  inherent  risks  the  toon  of  the  research  will  be  a  comoamcn  of  system* 
>wli.il;tlll  v  iwguuMMiuirti.  ond  |Miufc«.b*J  iwllalaAlv  with  actual  tuhluvutl  iwLiiuUy  of  'mI  ImI  sywbune. 

T)m  iwNorclr  k  IIiiiUkI  to  a  can  Mi  eecdo*  of  netur  crMcaf/ pacing  Iwiu  enJuiieil  n  If  mi  Quarterly  hudinwu  fbufiurtu 
•o  Congre-9>5  (HI *1  Tank,  AmphiWoi.^  Awioh  family  of  Vehicles.  5  Ton  Truck  family  of  Vehkr-ev  HMWVfV  family  of 
Vehicles,  ik|Ik  femotud  Family  of  Virilities,  tVS  fornlly  of  Wteike,  and  tfe-  MlC6  Howlt/es)  The  analysis  Is  llnutud  to 
an  assessment  of  rHlabuty  management  and  process  Issues  and  does  not  «peciflcn*r  address  commodity  or 
•erhonlngy  dilwen  rrihlbltv  prryrleme 


GENERAL  BACKGROUND  QUESTIONS 


1.  Product  Group: 
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S.  Current  Life  Cycle  Phase: 
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kata  Prediction  and  Deployment  fin  years)/ 


0.  What  wit  the  intended  We  cycle  of  the  weapon  system* 
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Hew  P.vc  2 


Pnfje  2  of  T 
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MANAGING  RELIABILITY 


7,  Rank  i>rii«»r  wti.it  you  cortviVr  loin*  tfiM  Tnp_Mv£>ri*lirtiiiliiv  nttvi.igunient  problem#  (I  being  the  moil 
severe  problem): 

n  Traditional  t®«  and  *«  ablation  RAM  metrics  a-o  nor  sc^gortod  bp  our  maincanar*:*  data  sourco;  <ur>atlo  to 
make  a  valid  comparison  between  RAM  recjjremente  arid  estmatea  with  actual  Khieved  Add  data) 

|  F.oliabilltv  Is  rot  a  Kay  Parfortnanc*  Parana bor  h  theORD 
|  Misting  or  poorly  witton  OFD  roiiotoiiitv  raqjiromoors 
|  MismMrpretancri  </  reliability  reietod  pec  formenc*  st>Eo:ir<aticne 

L  Ccritr-»:tcr  ret  -deigning  fet  relufoilty  auffic^nUy  *>c*re  the  reQjremert 
n  Acquisition  streamlining  and/or  government  dwneirng 
[I  Contractor  rot  using  t>»st  commardal  practices 
[  ”  Poor  reJleblity  c^amng-ard  grewth  planning  {test  too  latej 
|  Inadequate  or  poitita  are  guldanca  fnwi)  updating > 

|  Insufficiont  ratability  taarire  to  varify  raquranants 

|  Too  much  c raeaur  e  to  field  eystem  cheaper  (C«t  go-efe  outweigh  reliably) 

|  1  Too  much  cre-ajre  to  Add  tv  stem  mere  rep*dhr  (SchediJe  gods  outweigh  reieb'i'tv) 

[  Unrealistic  reliability  requremsnts  with  inadequate  rationale 
r  Naad  mere  qualified  personnel  h  rsilafcJIty  managoment  In  PM  offfc© 

|  Not  consistent^  improving  relUfelltty  after  fielding 

|”  Other  | - 3 


Additional  Commants:  f" 


(I  Who  within  your  organization  ■»  primarily  responsible  lor  reliability  activities  for  this  particular  program? 

<Chod<  only  one) 

r  Program  Manager  fPM>  r  Test  Team  Load* 

r  Logistics  Management  Specialist  iLMS)  C  Ratability  IPT  <lf  go,  Is  It  a  formally  charUrad  IPf?)  r  Yat  r  No 
f  Pro | act  Tiom  Uadsr  (TL>  <-  PrUno  Contractor 

C  3  pctams  Enqinaarriq  T  -rr  Lister  <“  Ho  ana  qoscir»:aliv 
^  Logbtlcs/Suppcrtab'ltv  Team  Leads-  <*  Other  ^ 


9.  How  in  the  system  notability  propnMn  »ww1  cxirretponrimg  ni.nwiyHim»i  .#uimmi  !•  formally  documented 
within  your  program?  (Please  then  only  the  primary  c^erridrg  document  i 
<“  Pliability  Program  Plan 
r  Contract  $0W 


c  TEMP  (Tetf  arvd  Equation  Waster  Plan> 


r  SAMP  {Single  Acquisition  Management  Plari> 
C  Np  formal  retiddiltty  msnage»n*rtt  piai 
r  other  {Please  Explain  > 

r~ —  — 1 


10.  Which  ocUvttlet  dld/do«ii  your  program  implement  to  recognize  and/or  evaluate  potential  failures  and 
Cmiim-h?  fFIr^e  (#>*.*<  that  «plv) 


r  Ooet  6tk)r.  Ti&sting 

r  Oeveict^nentel  T^triQ 
r  Crwirccmentel  'Stiese  Vcr«nmg<ES$> 
r  Pelablltv  Modelrg 


r  PeiiotHllTv  «?J8lin«tlOh  Test  (IQT) 
r  Process  F^l.ire  Mr<^-  ?r»J  EfTe-cte  Afiftlys* s  ''FfMEA) 
r  '.vshjll  Ana^yva 
r  Phyan^of  FeflueCPOFi 


tattpeu  www.nps.twvy  jiullMCAP>suivey.asp 
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PRalctiicv  AHocarbn 

r  r  ajlt  Tree  Anal,  sis 
r  PVobabllstfc  Fdljra  ASiaismgnt 

P  Failj-sj  Mcocs.  Efforts  arrt  Criticality  Anakas  fFMECA) 

or  FKEa 

P Reliability  Oe«ebcment',ljrc*rth  Te*t  iPD/GT} 


P  Fa  I  Ira  Rspctthp,  Anatftu,  and  Ccrrtcthe  Action 
<FFACAS1 

r  ('arts  Control  Program 

r  Otisr  |  3 

r  IX)  not  know 


11.  Are  you  aware  ot  any  specific  DO©  or  Marine  Corps,  policy /regulations  regarding  weapon  system 
reliability  management? 

f  Yea  (If  vm,  which  do  you  use  to  help  yoj  manage  I 

relatfllty?)  1  - 

c  no 

Oo  you  led  that  existing  policy  and  regulation*  on  reliability  provide  adequate  guidance?  Wea*e 
explain. 


r 


ADDRESSING  RELIABILITY  IN  THE  REQUIREMENTS  GENERATION  PROCESS 


12.  What  was  the  documented  reliability/ avail  ability  retirement?  in  what  term*  wax  it  measured 
MTBF,  MTBM,  A  M  IBSA,  M TBOMF ,  MTBEFF,  MTBOMA,  MTBMAF ,  etc)? 


r 


(•AS. 


13.  Was  a  reliability  requirement  identified  as  a  Key  Performance  Parameter  <KPP)  in  the  Oper ationai 
Requirement*  Document  (ORD)-'  If  »o,  wa*  the  requirement  in  an  objective  and  quantifiable  lorm  that 
contractors  and  the  government  could  easily  agree  upon?  Please  explain. 


14.  Was  tf»e  PMO,  as  the  Material  Oevefoper,  able  to  influence  Incorporation  ol  realistic  reliability 
requirement*  a*  part  ol  die  ORO  process?  Vcs  r  N«o  <“  Nlo t  S i^r»r 
Additional  Comments: 


COfiTRACJING  FOR  RELIABILITY 


15.  Wa*  rrhahtlity  included  as  a  factor  in  the  source  selection  process7 

r  Yes  (Wee  ,j  a  significant  chscanridt-x '  p  vea 

•"tfo 

AddtloralGcmmantt  jj 

15.  How  are  ORD  reliability  requirements  lor  your  program  translated  into  actual  contractual  reliability 
requirement*'/ 

r  CPC-  par  Wit  raiatfca  to  rAtiabiliti  ara  rastatao  in  S 0/i/:o*c  <I.a  cxitra:t  r*qjl-«m«srit  t;  to  ORD 
raqulnament) 

f'  AddUonal  le»ehof  relabiltv  r?  seeded  to  >he  contract  (Pieose  bdeOr  describe  the  ceocees) 

I - 3 


iittp  JJww  w.iifw.  txivy  .in  il'MCAP'JUxvty.asp 
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r  Ccmpreherave  ret  abilty  requirements  src  not  adojjatcly  stated  in  the  contract 
C  Other  (Explain) 

17.  Art*  there  incentives  employed  in  the  contract  that  are  specifically  tied  to  achieving  eye  tern  reliability 
performance  requirements? _ 

f  Yes  (Explain:  |  *jj 

r  No 

IB.  It  yes,  did  these  incentives  achieve  their  desired  effect? 

C  Yos  f*M5  ^  Too  early  to  tall 
AddttordlCdmrrwnte 


PEYELQPMENTAl  AMP  OPERATIONAL  TJESTIMC. 


19.  Did  the  user,  tester,  contractor  and  PM  office  ail  agree  upon  the  method  (model)  to  be  used  to 
determine  reliability  performance  during  testing? 

r  Vat  (If  yds,  i\hcr*  is  thrs  daojnfioc.tad?  TEMP?;,  f  3 

r  tto 

r  Geruin 
Addttorol  Ccmmants  | 


20.  Did  your  program  have  specific  1QTI  enfrance  criteria  relative  to  reliability* 

f  Yes  (please  provide  deteite) 
r  *3 

Addttohel  Comments  |  3 


21.  Was  the  amount  of  time  and  turning  allotted  for  rell Ability  testing  during  QT  sufficient  for  your  program' 

C  Yes  C  No  _ 

Addttonel Comments. 


22.  Did  the  system  eNperience  a  ivy  major  reliability  test  failures? 


r  Yes  f  No 
Adddohol  Ccmments  | 

RELIABILITY  REQUIREMENTS  VS.  CON  TRACTOR  ESTIMATES  VS.  ACHIEVED  RELIABILITY 


23.  To  what  level  w*«  your  system  *s  ORO  reliability  requirement  demonetnaiecf?  -j  t>  ter  n*.  cf  %  or CPD 

r-qjrements  m-t  bv  selecting  gr»»  frem  each  column) 


During  OT 

During  OT 

Durinq  £-4fitanrr%r  r 

r  ioo% 

r  100% 

r  ioo<* 

C  >40% 

<“  >«  % 

f  >40  % 

C  >87% 

e  >sa  % 

f  >80  % 

C  >70% 

f  >70% 

f  >70  % 

http:'.  iwww.nps.tti\y  jtul'MCAP.'Auvey  .asp  1  LQ7f20O2 
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25,  Was  Che  initial  contractor  reliability  estimate  documented-* 

I"  Yes  <*  No(  [f  yes,  on  wvbot  document?)  (” 


Is  such  documentation  retained? 

^  Yes  r  No  i  If  yes,  where  (physically)  and  by  whom?)  f 


C  >60% 

r  aeon 

c  >ea  % 

f  >so% 

f  >50% 

<"  >50% 

C  20-50% 

r  20-50% 

r  20-50% 

r  <20% 

O  <20% 

O  <20% 

r  Do  NOT  krew 

C  Do  NOT  krew 

C  Do  NOT  knew 

r  hoc  Applicable 

r  hcc Applicable 

C  hot  Applicable 

24.  To  what  level  was  the  contractor '*  reliability  estimate  demonstrated?  •  state  ir  te 

by  vr»ectngone  from  each  column) 

Durhcj  DT 

Cumci  OT 

Burro  -istarifTp^* 

C  100% 

O  100% 

C  1C0% 

r  >«  % 

C  >«!% 

r  >go‘h 

<-  >ki% 

f  >eo  % 

f  >80  % 

f  >70%. 

<*  >70% 

r  >70% 

r  >60% 

r  >60% 

r  >60% 

C  >50% 

C  >50% 

r  >5Cr% 

C  20-50% 

C  20-50% 

O  20-50% 

C  <20% 

r  <20% 

c  <20% 

r  Do  NOT  know 

C-  Do  NOT  krow 

I"  Do  NOT  know 

r  Not  APCHlSMt- 

f  Net  Apct 

f  not  A(ictic8tte 

26.  What  was  the  initial  (prime)  contractor  rchdbt  My  estimate  ter  the  system* 


In  what  terms  was  it  measured  (».e.  MTBF,  MTBM,  Art)? 


"2 


27.  Has  actual  aclweved  reliability  o4  the  Itelded  system  been  coBected/curnputed?  C  rgc  C  I,. 

If  so,  in  what  form  (i.e./  MTBF,  MTBM,  A#/  etc)? 

I - 3 

26.  What  has  been  the  overall  achieved  reliability  of  the  fielded  system  ler  its  entire  Ide  cycle  thus  lar" 

I - 2 


20.  How  Is  reliability  field  data  collected  to  gather  failure  and  repair  histones?  (Plea—  chock  all  that  aeph  T 
r  Intermediate  Si^ply.’Mantenerce  Activities  1“ Retiobihty  Data  is  Net  Formally  Collected 

r  Depot  or  CLS  Memtenarsre  Records  X~ Other  (Erpiahj 

r  usiro  Unit  Raadirissc  Rebirtrg 

30.  Does  current  achieved  field  reliability  data  indicate  your  system  meets,  exceeds,  or  has  failed  to  meet 
Use  DUD  reliability  requirement? 

f  Achieved  reliability  r*»  exceeded  OPO  reojtemerte 

r  iHiaCtlifv  has  met  0»D  rec|i»etner<B 

C  Achtov*d  reliability  he;  fallen  sheet  of  OPD  retirements 

fattp:'.>www.nps.isiN7  jnU'MCAP.<«uvey.asp  1  l/iTflOUi 
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Pape  fi  nf  2 


C  Rallebililf  data  rot  formally  ooltoctad 
<“  Raquiramarts  muanct  dainaated  r  tha-ORD 

Additional  comments  P"""  ! 


'Jl.  Doe*  current  achieved  field  reliability  data  indicate  your  system  meet*,  exceed*,  or  »va*  failed  to  meet 
the  contractor  reliability  estimate? 

C  Achis^ad  reliability  has  exceeded  contractor  esriir.aT*-; 
r  Achieved  reliability  ha? met  contractor  eetimares 
<*  Adi wv»d  ref labditv  has  fallen  shot  of  contractor  climates 
^  Reliability  data  not  formally  ooltocted 

C  Contractor  -eirima!**  haua  oat  bean  maintained  far  compariscn 
Additional  oonurent *■  | 

3?.  What  specific  organ!  ?acion(s),  if  any,  have  compared  and  Assessed  actual  achieved  reliability  of  fielded 
»y*tem*  to  the  original  contractor  e*tim ate*? 

I - 3 

If  completed,  was  it  done  a*  on  official  re%p©n*ibihty/rnle  assigned  by  higher  headquarter*  or  we*  the 
comparison  conducted  for  some  other  reason  (l.e.  internal  study)?  Please  explain. 

I  .  . . . .  a 


33.  Hhn  rhr  system  undergone  a  Oepcn  Level  MdinteriarH*  Pnigryan  <Si  FP/PIP/Pehuilil/IPOAN)? 

C  r  es  C  No  (3t  so,  which  one  and  at  what  stausVyear  h  Its  Its  cycle? ) 

I - 5 


34,  Following  any  Otpol  t  evH  Maintenance  Program,  Ims  the  adhiovexl  reliability  of  the  tyeieni  ifcv**t  fondly 
changed?  ^  Ves  f'  No  (If  ye*,  ptease  s> Cain) 


35.  Has  any  of  the  reliability  failure  data  collected  led  to  identification  of  OftS  cost  drivers  that  subsequent!- 
led  to  cost  effective  Improvements  ? 

71 

r  No 


Addittoral  comments 


3 


36.  I*  there  a  formal  reliability 

r  Yee  (If  where  is  it  doctfr*nt<*J>)  I  fl 

r  no  _ 

/■dclttonal  comments.  I 


37,  DOOft  your  system  miphiy  ,•  RHi.ihiliiv  Onnvd  M.ii'Ihi.cm  urogram? 

r  vos  (If  y*s,  ho*  isitformaly  bnpiamencatf?;i  ["" 

rwo  _ 

Addttorel  comments  [”" 


latp : ww  w.np*.  navy  jnil'MCAJV survey  .asp 


l  . 
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-<ir^  » 1  | 


Fee  rrore  'rfermaher.  contact  IC-aot  Marvin  Norcro^il  Dot*  i«t  edteC  1O/O3J02 


liltp :  |  ww  w.iifM.  t^ivy  .nul'MCAP' tuxvey  .ftsp 


1 1.Q7/2W2 
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APPENDIX  C.  MARINE  CORPS  MATERIEL  COMMAND 
MTBF/MAINTENAN CE  STUDY 


D0209  (MK-48)  MEAN  TIME  BETWEEN  FAILURES/MAINTENANCE  STUDY 

Assigned:  8  June  2002 
Completed: 

Completed  by:  Capt  Jake  Enholm 

Assisted  by  Jaeyong  Lee,  Assistant  Professor  of  Statistics,  Penn  State,  Maj  Humpert, 
Capt  Paige,  Capt  Frey,  CWO-2  Peterson,  Deborah  Whitley,  Mike  Everly,  Dennis  Cooper, 
GySgt  Pelligrin. 

Objective:  Formulate  a  methodology  for  determining  systemic  mean  time  between 
failures  and  equipment  parts  (NSNs)  failure  rates  using  MIMMS/ATLASSII/SASSY  data 
fields. 

Data  Used:  D0209  ERO  (Equipment  Repair  Order)  Data  from  1999-2001 
Number  of  Records:  34,592. 


INTRODUCTION 

This  study  was  conducted  in  order  to  determine  whether  mean  time  between 
failures  for  a  primary  end  items  using  current  warehoused  maintenance  management  data 
is  possible.  The  study  was  also  conducted  to  determine  whether  mean  time  to  failure  for 
a  particular  part  on  a  primary  end  item  using  the  same  data  is  possible. 

It  is  important  that  the  study  includes  the  reason  why  we  are  attempting  to 
calculate  mean  time  between  failures  for  equipment.  The  calculation  of  this  time  is 
necessary  in  order  to  determine  whether  the  mean  time  between  failures  is  increasing, 
decreasing,  or  remaining  constant  with  age.  As  equipment  ages,  its  mean  time  between 
failures  decreases  until  the  cost  of  keeping  that  item  operational  is  more  than  the  cost  of 
buying  a  new  item.  Estimates  of  when  maintenance  costs  will  exceed  acquisition  costs 
are  questionable  without  mean  time  between  failures  calculation.  Each  piece  of 
equipment  we  are  concerned  with  is  a  system  of  working  parts,  and  we  ref  er  to  its  failure 
as  a  systemic  failure.  A  complete  estimate  of  maintenance  costs  should  concentrate  more 
on  mean  time  between  maintenance  instead  of  mean  time  between  failure  in  order  to 
capture  all  costs. 

Mean  mileage  between  failures  vice  mean  time  between  failures  might  be  a  better 
indicator  for  item  replacement  in  some  equipment,  such  as  trucks.  It  was  established  in 
an  earlier  study  that  the  MK-48  meter  reading  field  of  the  Marine  Corps  Integrated 
Maintenance  Management  System  (MIMMS)  and  ATLAS  S  II  data  that  records  odometer 
information  was  unusable  for  most  records  (Enholm,  2002).  It  might  be  possible  to 
construct  a  sophisticated  algorithm  to  determine  a  mean  mileage  for  the  fleet,  but  the 
nature  of  the  current  study  requires  lower  error  rates  than  are  currently  found  in  the  meter 
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reading  field.  It  is  this  reason  why  mean  mileage  between  failures  cannot  be  accurately 
determined  at  this  time. 

The  calculation  of  mean  time  to  failure  for  a  part  in  a  system  is  important  in  order 
to  statistically  isolate  a  problem  area.  Mean  time  to  failure  calculation  is  also  important 
when  comparing  the  amount  of  time  we  can  expect  a  part  to  be  in  operation,  compared  to 
its  cost. 

In  the  calculations  used  in  this  study,  mean  time  between  failure  (MTBF)  and 
mean  time  between  maintenance  (MTBM)  periods  were  combined.  The  addition  of  the 
deadline  control  date  field  can  be  used  to  tell  if  a  vehicle  is  deadlined  or  just  operating  in 
a  degraded  fashion.  The  deadline  control  date  field  is  left  blank  on  equipment  repair 
orders  (EROs)  that  are  not  deadlined.  This  study  substitutes  median  time  between 
failures/maintenance  for  mean  time  when  discussing  alternate  methodologies,  i 

SYSTEMIC  MEAN  TIME  BETWEEN  FAILURE/MAINTENANCE 

The  calculation  of  a  time  to  failure/maintenance  of  an  item  requires  that  we  know 
the  time  an  item  has  been  in  operation.  For  systemic  failure  or  maintenance  degradation 
there  is  also  a  requirement  that  the  age  of  the  system  be  established.  This  is  particularly 
important  when  attempting  to  establish  the  age  when  an  item  starts  spending  longer  and 
longer  times  down  as  it  gets  older,  and  maintenance  starts  adversely  affecting  operations. 

Unfortunately,  there  is  not  a  central  repository  or  data  warehouse  where  the 
Marine  Corps  has  information  establishing  the  age  of  a  serialized  item.  The  program 
managers  of  an  item  keep  a  logbook  of  serial  numbers  that  detail  when  a  serial  number 
was  fielded.  This  information  is  not  always  in  electronic  format,  or  readily  available. 
For  this  particular  study,  information  fom  MARCORSYSCOM  helped  establish  the 
number  of  MK-48s  that  were  fielded  between  the  years  1985  and  1999. 


1985 

287 

1986 

354 

1987 

434 

1988 

574 

1989 

31 

total 

1680 

Table  1.  MK-48  Fielding  Numbers  by  Year. 

These  numbers  were  used  to  estimate  the  year  a  serial  number  was  fielded. 
Estimation  had  to  be  done  in  this  case  because  no  database  was  available  that  specifically 
contained  this  information. 

Serial  numbers  are  given  sequentially  as  an  item  is  fielded.  Using  the  number  of 
MK-48s  fielded  each  year  combined  with  the  serial  numbers  found  in  the  ERO  header 


1  Statistically,  this  is  not  the  same  number.  The  sample  median  time  between  failures  is  a  point  in  time 
where  50%  of  the  samples  fail.  The  sample  mean  time  is  the  average  time  between  failures.  Because  we 
cannot  always  calculate  the  mean  reliably  when  using  certain  statistical  methods,  we  will  sometimes  have 
to  substitute  a  median  as  the  most  reliable  substitute. 
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records,  a  table  was  constructed  that  estimated  the  ages  of  the  serial  numbers  from  1985  - 
1989. 


Serial 

Yr 

Number 

Fielded 

515814 

1985 

515842 

1985 

516137 

1985 

516188 

1985 

516276 

1985 

516367 

1985 

Table  2.  A  Portion  of  the  MK-48  Serial  Numbers  -  Year  Fielded  Table, 

These  were  estimated  by  sorting  the  serial  numbers  found  in  maintenance 
management  data  and  counting  them  out  according  to  the  totals  in  Table  1. 

The  maintenance  cycle  of  an  item  such  as  the  MK-48  is  commonly  referred  to  as 
a  repairable  process  because  the  item  transitions  between  operating  and  non-operating 
status.  The  model  used  here  is  a  maintenance  model,  where  both  preventive  and 
corrective  maintenance  actions  are  applied  to  systems  being  studied.  Leemis  (1997) 
advocates  the  use  of  a  non -homogeneous  Poisson  process  to  model  failures  for  repairable 
systems  since  it  can  model  deteriorating  systems.  Non-homogeneous  means  that  the  time 
between  failures  can  increase  or  decrease.  For  a  definition  of  a  Poisson  process,  see 
Appendix  A. 

Once  the  age  of  a  vehicle  has  been  established,  the  time  between  failures  can  be 
correlated  with  the  age  of  the  vehicle  if  a  wide  enough  data  range  is  established. 
Unfortunately,  although  the  Marine  Corps  has  maintenance  management  data 
warehoused  back  to  1998,  the  data  from  1998  appears  to  be  incomplete.  There  were 
between  8000-13000  equipment  repair  order  (ERO)  records  recorded  for  MK-48s  from 
1999-2001.  But  in  1998  there  were  only  1400  records,  suggesting  that  the  warehousing 
was  incomplete  and  the  1998  data  should  not  be  used.  This  assumption  reduced  our 
available  data  range  to  1999-2001. 

Systemic  failures  were  compiled  using  the  assumption  that  a  group  of  EROs  with 
a  common  serial  number  and  the  same  date  received  in  shop  (DRIS)  entry  consists  of  one 
job  for  a  particular  vehicle.  The  time  between  DRIS  and  the  date  the  job  was  closed  (all 
the  EROs  for  that  job  are  closed  out)  is  also  of  particular  value.  The  number  of  days  the 
vehicle  was  worked  on  for  a  particular  job  reduced  the  amount  of  time  that  the  vehicle 
was  available.  In  general,  if  two  EROs  have  overlapping  dates,  the  dates  that  cover  the 
largest  period  of  time  are  used  for  failure  calculation. 


CENSORED  AND  UNCENSORED  DATA 

The  use  of  data  from  1998-2002  involves  the  use  of  censored  data.  Censored  data 
involves  the  use  of  incomplete  amounts  of  time  in  some  samples  because  our  records 
stopped  at  a  particular  date.  If  we  have  a  serialized  piece  of  equipment,  such  as  MK-48 
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515814,  and  we  know  when  in  2001  it  was  worked  on,  but  we  didn’t  see  it  worked  on 
again  before  we  ended  our  study  in  2002,  then  we  say  our  data  is  right  censored  for  that 
period  of  time.  If  we  have  another  serial  number,  515842,  and  we  know  it  was  worked 
on  once  in  2001,  and  again  in  2002,  then  we  know  the  amount  of  time  that  serial  number 
was  running  before  it  was  worked  on  again.  We  say  that  that  sample  was  uncensored. 
Both  censored  and  uncensored  observations  provide  clues  to  generating  a  mean  or 
median  time  between  failure/maintenance.  The  statistical  calculations  were  done  using 
Kaplan-Meier  survival  estimates,  which  estimate  the  probability  that  an  item  will  survive 
to  a  particular  time  by  conditioning  on  the  probability  that  it  survived  up  to  the  previous 
period.  For  more  information,  see  Kaplan  and  Meier,  1958. 

Kaplan-Meier  survival  statistics  were  a  useful  tool  to  describe  the  MK-48’s 
failure/maintenance  periods.  Figure  1  shows  the  Kaplan-Meier  (K-M)  survival 
probabilities  for  operating  a  MK-48  up  to  a  particular  day  based  upon  maintenance 
management  system  data  from  1999.  Data  calculation  for  a  median  for  1999  data  was 
attempted,  to  see  if  a  distinct  median  for  every  year  of  warehoused  data  could  be  isolated. 
If  so,  we  could  see  if  the  medians  for  several  years  were  non-homogenous.  The  gaps  on 
the  right  part  of  the  graph  show  the  censored  data  observations.  A  median  time  to  failure 
could  not  be  calculated  because  there  were  too  many  censored  observations. 


K-M  Survival  Probability  and  Gaps  for 
1986  MK-48s  in  1999 
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Figure  1.  Kaplan-Meier  survival  probabilities  show  ing  the  probability  that  a  MK-48 
fielded  in  1986  will  operate  up  to  a  certain  day  before  needing  maintenance.  The  data 
used  was  from  1999  maintenance  EROs.  Data  that  was  censored,  or  incomplete  shows 
up  as  gaps.  The  large  amount  of  censored  data  did  not  allow  a  median  to  be  calculated. 

Because  a  median  could  not  be  calculated  using  just  the  1999  data,  the  data  period 
was  widened  to  include  1999-2001.  Table  3  shows  the  results  of  that  analysis. 


136 


Percentiles  of  (all  years) 

the  Survival  Function 

Survival 

Time 

25'th  percentile  (lower  quartile) 

87 

50'th  percentile  (median) 

181 

75'th  percentile  (upper  quartile) 

mi 

Table  3.  Analysis  output  from  the  software  program  Statistica  showing  Kaplan  - 
Meier  survival  statistics  for  MK-48s,  using  maintenance  management  data  from  1999- 
2001.  The  statistics  reflect  a  median  time  to  failure/maintenance  of  181  days. 

The  median  time  between  failure/maintenance  of  MK-48s  of  181  days  seemed 
excessive  given  current  readiness  rates,  however  it  must  be  taken  into  account  that  this 
rate  includes  all  MK-48s  in  the  Marine  Corps.  The  Mk-48s  in  stores,  and  on  Maritime 
Pre- positioning  Force  (MPF)  ships  might  be  affecting  this  trend.  Acquisition  of  an 
additional  data  set  that  filters  out  stores  and  MPF  ship  MK-48s  is  required  to  test  this 
hypothesis. 

It  was  also  possible  to  calculate  the  median  time  between  failures/maintenance  for 
a  group  of  MK-48s  fielded  in  a  particular  year.  The  expected  median  times  between 
failure/maintenance  should  be  decreasing  with  age.  Figure  2  shows  that  this  is  indeed  the 
case,  with  the  exception  of  MK-48s  that  came  out  in  1987.  (This  excludes  vehicles 
fielded  in  1989  because  the  low  number  of  vehicles  fielded  in  that  year) 


MK-48  Median  K-M  Survival  Times 


Year  Vehicle  Fielded 


Figure  2.  Median  K-M  times  between  failure/maintenance  correlated  w  ith  estimated 
age.  The  median  times  decrease  with  age,  with  the  exception  of  vehicles  fielded  in  1987. 
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Year  of  MK-48  and  MTBM 


Year  MK-48  Fielded 


Figure  3.  Mean  times  between  failure/maintenance  correlated  with  estimated  age 
using  the  average  observed  times  and  discarding  the  censored  observat  ions.  The  Kaplan  - 
Meier  method  was  not  used  here.  The  mean  times  do  not  decrease  with  age,  but  the 
values  are  more  reasonable  given  current  MK-48  readiness  rates. 

An  alternative  method  to  the  K-M  methodology  was  also  used.  This  methodology 
discarded  the  censored  observations  and  used  only  samples  that  contained  two  jobs  or 
more.  The  amount  of  time  the  vehicle  was  operational  between  jobs  for  a  similar  serial 
number  was  calculated.  This  time  was  then  averaged  for  all  observations.  The  sample 
mean  for  this  method  was  41.2  days.  The  mean  seems  more  reasonable  with  current 
readiness  levels  for  the  fleet,  but  this  methodology  did  not  show  increasing  MTBF/M  for 
increasing  age. 

A  possible  validation  of  tying  in  a  set  of  EROs  with  a  similar  serial  number  and 
the  same  DRIS  date  as  one  job  was  seen.  The  jobs  were  correlated  with  the  amount  of 
time  each  vehicle  was  worked  on.  An  average  readiness  for  the  fleet  was  calculated 
using  the  amount  of  time  worked  on  for  each  job.  This  calculation  was  adjusted  for  a 
data  period  that  corresponded  with  archived  MK-48  readiness  in  the  Material  Readiness 
Assessment  Module  (MRAM).  The  job -methodology  calculated  readiness  was  82%. 
The  MRAM  readiness  for  the  same  period  of  time  was  also  82%.  It  should  be  noted  that 
the  MRAM  only  uses  deadlined  vehicles,  and  the  data  used  in  this  study  was  for  all 
maintenance  tasks.  Further  calculation  with  another  primary  end  item  is  necessary  before 
any  conclusions  can  be  made  on  this  validation  process. 

NSN  TIME  TO  FAILURE 

The  equipment  repair  order  records  were  “drilled  down”  or  linked  to  National 
Stock  Numbers  (NSNs)  that  were  ordered  for  a  particular  ERO.  This  would  not  have 
been  possible  to  compile  in  a  short  period  of  time  without  the  help  of  the  integrated 
SCOPE  database  constructed  by  Capt  Paige’s  team.  The  necessary  data  was  compiled 
with  the  integrated  system  in  three  minutes.  A  parallel  effort  using  the  conventional 
maintenance  management  record  data  system  in  place  required  two  weeks. 
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Calculation  of  a  mean  time  to  failure  of  an  NSN  was  done  under  the  assumption 
that  the  NSN  failed  and  another  was  ordered  and  replaced  the  failed  item.  This  did  not 
include  the  repair  cycle  used  in  secondary  repairable  items  (secreps)  such  as  engines  and 
transmissions.  Calculation  of  a  mean  time  to  failure  for  an  NSN  is  problematic  because 
there  might  be  several  NSNs  that  are  in  use  for  a  particular  system  that  perform  the  same 
function. 

A  common  statistical  method  for  modeling  lifetimes  of  equipment  parts  is  the 
Weibull  distribution,  which  is  a  “...generalization  of  the  exponential  distribution...” 
(Leemis,  p.  88).  The  Weibull  distribution  was  selected  for  modeling  two  of  the  most 
common  NSNs  found  in  the  MK-48  maintenance  records.  Table  4  is  a  partial  printout 
from  the  statistics  software  program  Statistica  and  shows  parameter  estimates  that 
Statistica  came  up  with  after  looking  at  the  data  from  the  selected  NSN.  The  low  p- 
values  (also  known  as  observed  significance  levels)  on  the  right  indicate  that  the  data 
does  not  fit  the  Weibull  distribution  well.  The  NSN  chosen  was  one  of  the  most 
frequently  encountered  ones  in  the  data  set.  Figure  4  shows  the  survival  probability 
distribution  for  these  samples. 


Parameter  Estimates,  Model:  Weibull  (nsnl) 

Note:  Weights:  1=1.,  2=1./V,  3=N(I)*H(I) 

Variance 

■itd.  Err. 

Lambda 

Lambda 

Lambda 

Chi-Sqr. 

if 

P 

Weight  1 

0.000662 

3.2E-07 

0.000565 

18.33028 

9 

0.031563 

Weight  2 

0.000747 

2.09E-07 

0.000457 

19.10768 

9 

0.024322 

Weight  3 

0.001918 

2.01E-06 

0.001417 

19.23488 

9 

0.023297 

Table  4.  Statistica  parameter  estimates  for  a  MK-48  Weibull  lifetime  distribution. 
The  low  p- values  indicate  a  low  level  of  confidence  that  the  NSN  distribution  fits  a 

Weibull  curve. 


MK-48  NSN  2510012331768  Cummulative 
Proportion  Surviving 
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Figure  4.  Weibull  probability  of  NSN  2510012331768  surviving  up  to  a  particular 

day. 
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A  second  MK-48  NSN  that  was  frequently  ordered  was  analyzed.  The  high  p- 
values  that  Statistica  generated  indicate  that  the  data  was  close  to  a  Weibull  distribution 
with  the  chosen  parameters.  Statistica  gave  this  NSN  a  median  life  time  of  1118  days. 
Figure  5  shows  the  survival  probabilities  for  the  parameters  in  Table  5. 


Parameter  Estimates,  Model:  Weibull  (nsn2) 

Note:  Weig 

hts:  1=1.,  2=1./V,  3=N(I)*H(I) 

Variance 

Std.Err. 

Variance 

Lambda 

Lambda 

Lambda 

'iamma 

Gamma 

Chi-Sqr. 

df 

P 

Weight  1 

0.000791 

1.07E-06 

0.001035 

0.864681 

0.040519 

6.143479 

9 

0.725468 

Weight  2 

0.001402 

1.37E-06 

0.001171 

0.789683 

0.015711 

5.995469 

9 

0.740362 

Weight  3 

0.001913 

2.82E-06 

0.001679 

0.728864 

0.017097 

6.031236 

9 

0.736779 

Table  5.  Statistica  parameter  estimates  for  a  MK-48  Weibull  lifetime  distribution. 
The  high  p- values  indicate  a  high  level  of  confidence  that  this  NSN  distribution  fits  a 

Weibull  curve. 


Figure  5.  Weibull  probability  of  NSN  2540012348073  surviving  to  a  particular  day. 

The  median  (not  the  mean)  number  of  days  for  survival  was  1118  days.  This  number 
indicates  the  curve’s  downward  trend  is  steeper  after  1118  days. 

CONCLUSIONS  AND  RECOMMENDATIONS 

The  mean-time  between  failure/maintenance  analysis  of  the  MK-48  revealed 
some  areas  where  maintenance  management  data  systems  can  be  improved.  The 
improvements  will  make  this  type  of  analysis  more  accurate,  and  more  useful  with 
respect  to  estimating  when  a  system’s  mean  time  between  failures/maintenance  is 
increasing  or  decreasing  with  age.  The  study  produced  two  different  methodologies  to 
calculate  time  between  systemic  maintenance/failure.  The  analysis  of  MK-48  NSNs  and 
their  median  time  to  failure  also  revealed  some  areas  that  can  be  beneficial  to  producing 
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useful  results. 

The  following  recommendations  would  increase  the  accuracy  of  systemic  and  NSN 
failure  rate  calculation: 

1)  A  database  containing  primary  end  item  serial  numbers,  the  year  fielded,  and  their 
cost  needs  to  be  developed.  A  second  database  where  the  NSNs  of  a  primary  end 
item,  their  description,  and  their  cost  needs  to  be  constructed.  NSNs  that  perform 
the  same  function  need  to  be  cross  -referenced. 

2)  The  current  warehoused  maintenance  data  of  the  Marine  Corps  needs  to  be 
extended  back  in  time  as  far  as  possible. 

3)  Perhaps  the  best  alternative  for  error  checking  of  serial  numbers  will  be  provided 
with  the  implementation  of  the  Global  Combat  Support  System -Marine  Corps 
(GCSS-MC).  The  GCSS-MC  system  will  replace  our  current  legacy  maintenance 
systems  and  could  contain  serialized  records  for  each  primary  end  item  in  the 
Marine  Corps.  If  a  manpower  -type  record  of  information  for  a  serial  number  can 
be  checked  when  a  new  ERO  is  entered,  a  calculation  can  be  done  to  see  if  the 
new  entry  makes  sense.  A  sophisticated  algorithm  known  as  an  intelligent-agent 
can  run  through  a  series  of  decision  trees  that  look  at  past  dates  and  entries  for 
meter  readings  for  that  serial  number.  The  intelligent  agent  then  makes  a  decision 
whether  or  not  a  meter  reading  is  reasonable  for  that  serial  number.  If  not,  a 
notification  back  to  the  Maintenance  Management  Officer  of  the  unit  that  made 
the  entry  can  be  sent  with  a  request  for  clarification  of  the  new  entry. 

It  is  possible  to  calculate  a  sample  mean  or  median  time  between 
failures/maintenance  for  some  of  the  equipment  in  the  Marine  Corps  using  the 
methodologies  presented  in  this  study.  The  accuracy  of  such  analysis  will  be  suspect  if 
the  current  weaknesses  of  the  system  are  not  fixed.  A  validation  of  the  most  accurate 
method  is  currently  being  conducted. 


APPENDIX  A 

Ross  defines  a  Poisson  process  as  a  “The  counting  process  {N(t),  /=()}  is  said  to 
be  a  Poisson  process  having  rate  ?,  ?  >  0,  if 

i.  MO)  =  0 

ii.  The  process  has  independent  increments 

iii.  The  number  of  events  in  any  interval  of  length  t  is  Poisson  distributed  with 
mean  ?t.  That  is,  for  all  s,  t  =  0 


P{N(t  +  s)-N(s)  =  n}  =  e-h  —t,  n  =  0,1..." 

n\ 
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Notes  on  Julian  Dates 


A  challenge  encountered  was  the  way  dates  are  entered  into  EROs.  At  the  time 
MIMMS  was  implemented  there  were  many  reasons  why  a  Julian  dating  system  was  used 
for  date  entries.  In  the  year  2002,  the  system  is  a  hindrance  and  not  necessary.  The  way 
the  system  works  now,  the  mechanic  takes  a  standard  date  and  with  the  use  of  a  Julian 
date  calendar  converts  the  date  and  inputs  it  into  the  system.  Then  the  individual  looking 
at  the  data  looks  at  the  Julian  date  and  converts  it  back  into  a  standard  format  that  is 
understandable. 

The  Julian  date  format  is  also  problematic  when  using  it  in  Excel  or  Access,  the 
two  most  common  forms  of  data  manipulation  software.  Excel  c  an  calculate  the  number 
of  days  between  two  dates  by  simply  subtracting  the  two  date  in  standard  month/day/year 
format.  Excel  automatically  does  the  rest.  When  the  date  is  in  Julian  format,  string 
extraction  functions  must  be  used  that  convert  the  field  into  standard  month/day/year  and 
then  the  calculation  can  be  performed.  The  strings  that  the  Julian  dates  are  stored  in  are 
also  problematic.  Excel  has  a  problem  properly  sorting  these  strings. 


MTBF/M  FORMULATION 

Indices 

j  job  number 

y  year  of  job 

s  serial  number  of  equipment 

Variables 

aj_y  s  The  date  that  an  ERO  or  group  of  EROs  in  a  job  was  opened.  The 

date  the  group  of  EROs  was  opened  should  be  the  earliest  date 
received  in  shop  (DRIS)  in  the  group.  Job  j  in  of  equipment  in  year 
y  with  serial  number  ,v. 

Ojy,}S  The  date  that  an  ERO  or  group  of  EROs  in  a  job  was  closed.  The 

date  the  group  of  EROs  was  closed  should  be  the  latest  date  in  the 
group.  Job  j  in  of  equipment  in  year  y  with  serial  number  s. 

Dj-JiS  The  number  of  days  between  jobs  j  and  j  +  1. 

Cj,a,y,s  Censored  time  for  job  j,  in  year  y,  for  serial  number  s.  This  is  the 

time  from  the  date  the  job  was  closed  to  the  end  of  the  data  taking 
period. 

Ns  Total  Number  of  Jobs  for  a  serial  number. 
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L  Total  number  of  serial  numbers  in  equipment  being  analyzed. 

8  Censoring  indicator:  1=  censored,  0  =  uncensored. 

Ts  Survival  time  observation  for  serial  number  s. 

Formulation 


Dj’.y.s  - 

Xs  = 

T  =m  i  n(  X,  C) 

5  =  T(X<C) 


aj+l,y,s~  Oj.y.s 

rvN-l 


Calculation  of  the  number  of  days  between  jobs 

Calculation  of  mean  number  of  days  per  job  for  a 
particular  serial  number. 

The  minimum  value  between  X  and  C  is  picked  for 
T. 

If  X  is  less  than  C  then  8=0,  else  8=1. 


Two  methods  were  used  for  comparison  of  systemic  mean  time  between 
failures/maintenance  in  this  study: 


1)  \  =  Kaplan- Meier  (K-M)  product  limit  estimate  for  T  =min(X,Q  and  8  = 

I(X<C) 

2)  Aq  =  l/Xs  =  The  inverse  of  the  average  days  between  jobs  for  the  observed 
samples.  This  methodology  discards  the  censored  observations. 


1/  A,0  Mean  Time  Between  Maintenance/Failures  for  equipment  with  age 

a.  (Method  2  only) 

Notes  from  calculating  systemic  Median  Times  Between  Failures/Maintenance: 

1 )  Averaging  the  number  of  days  between  jobs  on  the  same  serial  number  does  not 
calculate  points  from  censored  observations  that  might  result  from  the  date  that 
the  last  job  was  closed  until  the  end  of  the  data  period.  These  points  are  not  taken 
into  account. 

2)  The  survival  data  generated  with  a  Kaplan-Meier  distribution  implies  that  the 
missing  observations  need  to  be  extended  by  increasing  the  warehousing  of  data 
back  in  time  in  order  to  get  this  data.  A  median  can  be  calculated  from  the  current 
data  set,  but  it  would  be  better  to  get  a  median  from  each  year  in  the  data  set. 
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APPENDIX  B 


Notes  from  calculating  part  (NSN)  failures: 

1)  NSNs  were  calculated  by  separation  from  the  main  data  set  and  sorted  by  serial 
number  of  vehicle  they  were  mounted  on. 

2)  If  an  NSN  was  mounted  on  the  same  vehicle  two  times  or  more  the  number  of 
days  it  lasted  before  another  NSN  with  the  same  number  was  mounted  on  it  was 
used  as  an  observation. 

3)  If  an  NSN  was  mounted  on  a  vehicle  and  left  there  until  the  data  period  ended  we 
count  the  number  of  days  between  the  date  the  job  was  closed  and  the  date  the  last 
piece  of  data  was  collected.  This  data  is  annotated  as  being  right  censored. 

4)  What  if  the  part  wasn’t  the  same  one  that  was  mounted  before?  For  example  we 
replaced  the  right  headlight,  then  the  left  headlight  goes  out  and  we  replace  that? 
We  have  no  way  of  telling. 

5)  What  if  the  NSN  was  replaced  by  a  different  NSN  for  some  reason?  We  don’t 
know  this,  but  it  can  be  fixed  with  an  NSN  database. 
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